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Pre  f  ac e 


The  purpose  of  this  research  effort  was  o  develop  a 
cost-risk  assessment  method  that  incorporated  the  effects  of 
cost  dependence  among  system  components.  Current  cost-risk 
assessment  techniques  make  limiting  assumptions  of  component 
cost  independence  or  total  component  cost  dependence, 
neither  of  which  we  felt  truly  represented  system 
relationships. 

The  method  we  developed  and  tested  provides  insight 
into  the  modelling  of  component  cost  dependency  and  its 
application  to  cost-risk  assessment.  Further  research  in 
this  area  will  improve  the  ability  of  the  Department  of 
Defense  to  estimate  cost-risk  and  to  limit  weapon  system 
cost  growth. 
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Abstract 

This  research  project  developed  a  cost-risk  assessment 
method  that  incorporated  the  effects  of  cost  dependency 
between  components  in  a  system.  The  method  uses  program 
personnel's  subjective  assessments  of  component  dependency 
as  inputs.  A  simulation  model  was  developed  and  employed  to 
test  the  method  under  various  levels  of  component  dependence 
strength  and  direction,  estimation  error,  and  system  size. 

The  analysis  was  accomplished  by  performing  sensitivity 
analysis  on  the  predictive  capabilities  of  the  cost-risk 
assessing  method.  Results  Indicate  that  the  method  has 
strong  predictive  capability  when  component  size  is  small 
and  when  the  direction  of  the  component  dependencies  is 
mixed.  It.  was  also  determined  that  the  use  of  component 
dependency  assessments  produced  more  realistic  total  system 
variances  than  those  produced  under  the  assumption  of 
component  cost  i nde pond  one o . 


AN  EVALUATION  OF  COMPONENT 
DEPENDENCE  IN  COST-RISK  ASSESSMENT 

I .  Introduction 

General  Issue 

The  basic  economic  problem  In  the  world  is  scarcity. 
Scarcity  in  critical  human  skills,  raw  materials,  products 
and  services  characterizes  an  environment  of  limited  re¬ 
sources.  Like  all  other  nations,  the  United  States  must 
operate  within  its  resource  constraints.  Resource  scarcity 
pervades  and  influences  decisions  at  all  levels  of  our 
government  and  industry. 

A  Macro  View .  Productive  resources  are  distributed 
between  the  competing  demands  of  the  public  and  private 
sectors  of  the  United  States  economy.  Furthermore,  the  sum 
total  of  resources  available  for  public  and  private  use  is 
generally  considered  to  be  fixed  in  the  short  run.  Conse¬ 
quently,  the  Government’s  share  of  our  nation's  scarce  re¬ 
sources  can  only  be  increased  at  the  expense  of  private 
consumption  and/or  investment.  Even  when  there  are  unem¬ 
ployed  resources  available,  the  utilization  of  these  re¬ 
sources  in  the  public  sector  may  still  have  a  negative 
impact  on  the  private  sector  if  they  are  paid  for  with  tax 
dollars , 

Within  the  public  sector,  the  government  must  allocate 
human  and  material  resources  to  fulfill  both  national  de- 


fense  and  social  responsibilities.  Increasing  our  defense 
capability  by  producing  the  B-1B  Bomber  or  the  Peacekeeper 
missile  requires  either  a  cutback  in  the  resources  allocated 
to  social  programs,  or  a  shifting  of  resources  from  the 
private  to  the  public  sector.  In  addition,  political  deci¬ 
sions  in  a  democratic  society  cannot  deviate  substantially 
from  the  public  will  without  a  loss  of  public  trust.  Conse¬ 
quently,  the  dual  constaints  of  public  will  and  scarcity 
limits  both  the  quantity  of  resources  allocated  to  the 
public  sector  and  their  distribution  among  competing  public 
needs. 

Since  money  is  used  by  free  market  societies  as  a 
medium  for  valuing  resources,  the  government's  allocation  of 
its  human  and  material  assets  is  accomplished  through  the 
budget  process.  The  government  indicates,  through  its  ap¬ 
propriation  of  public  funds,  how  it  wants  resources  divided 
between  defense  and  social  programs.  The  government's  re¬ 
sponsibility  to  see  that  these  funds  are  wisely  spent  is 
ingrained  in  the  words  of  Thomas  Paine: 

Public  money  ought  to  be  touched  with  the  most 
scrupulous  consciousness...  It  is  no.  the  produce  of 
riches  only  but  of  the  hard  earnings  of  labor  and 
poverty.  It  is  drawn  even  from  the  bitterness  of 
want  and  misery.  Not  a  beggar  passes,  or  perishes  in 
the  streets,  whose  mite  is  not  in  that  mass  [22:1], 

As  u  steward  of  public  funds,  the  DoD  shares  the  same  ethi¬ 
cal  responsibility  common  to  all  government  agencies  in 
expending  those  funds. 

Another  reason  the  military  establishment  must  use  its 


resources  effectively  and  efficiently  is  to  gain  and  main¬ 
tain  credibility.  Since  Congress  determines  what  portion  of 
the  federal  budget  is  to  be  appropriated  for  national  de¬ 
fense,  Congress  must  feel  confident  that  DoD's  funds  are 
wisely  used.  The  level  of  DoD  appropriations  is  also  indi¬ 
rectly  influenced  through  the  attitudes  of  Congressional 
constituents.  When  the  public  supports  defense  programs,  it 
is  easy  for  'pro-defense1  3-'islators  to  vote  their  con¬ 
science.  Even  those  who  traditionally  favor  social  programs 
will  be  more  compelled  to  support  defense  activities  in 
order  to  maintain  favor  with  their  constituencies.  For 
these  reasons,  the  military  must  maintain  a  strong  public 
image,  based  on  the  efficient  use  of  resources  to  meet 
essential  goals.  To  do  otherwise  would  undermine  its 
ability  to  secure  the  resources  necessary  to  provide  for  a 
strong  national  defense. 

The  Micro  View .  Resource  scarcity  influences  decisions 
at  all  levels  of  government;  the  Department  of  Defense  is  no 
exception.  Resource  limitations  create  controversy  con¬ 
cerning  the  correct  combination  of  programs  to  maximize  our 
military  capability.  What  is  the  optimal  blend  of  strategic 
and  tactical  forces?  Must  the  purchase  of  new  weapon  sys¬ 
tems  be  sacrificed  to  retain  manpower  at  desired  levels? 

These  are  but  a  few  examples  of  trade-off  decisions  that  DoD 
managers  continually  make  while  pursuing  our  defense  goals. 
Annual  Program  Objective  Memorandum  (POM)  decisions  reflect 
the  internal  competition  within  the  DoD  for  scarce  resources. 


Resource  scarcity  critically  impacts  the  way  DoD  de¬ 
velops  and  procures  its  weapon  systems.  With  an  emerging 
emphasis  on  readiness  and  maintainability  issues,  a  larger 
share  of  future  funding  will  likely  be  diverted  from  those 
who  acquire  systems  to  those  who  operate  and  maintain  them. 
Even  within  the  acquisition  community,  there  are  competing 
resource  demands.  Trade-offs  are  frequently  made  between 
the  amount  of  development  effort  and  the  number  of  systems 
we  produce.  Lack  of  adequate  funding  can  often  limit  de¬ 
velopment  effort,  in  areas  such  as  test  and  evaluation,  in 
order  to  maintain  desired  production  quantities.  Another 
example  of  resource  trade-offs  is. the  'stretching  out'  of 
efficient  production  schedules  to  redirect  funds  to  com¬ 
peting  programs.  These  examples  clearly  indicate  that  the 
distribution  of  scarce  resources,  a  problem  at  all  levels  of 
our  economy,  forces  major  trade-offs  in  the  acquisition  of 
weapon  systems. 

DoD  resource  trade-offs  are  couched  in  an  environment 
of  imperfect  information.  A  by-product  of  this  uncertainty 
is  cost  growth  in  weapons  acquisition.  Cost  growth  is  the 
increase  from  the  initial  program  cost  estimate  to  the  final 
program  cost  (3:105).  Clearly,  not  all  cost  growth  can  be 
viewed  in  a  negative  context.  Cost-effective  specification 
changes,  for  example,  often  enhance  weapon  system  perform¬ 
ance  beyond  the  initial  design.  Likewise,  boosting  produc¬ 
tion  quantities  is  also  an  intentional  management  decision 
to  increase  both  cost  and  defense  capability.  It  would  be 


naive,  however,  to  imply  that  all  cost,  growth  is  controlled 
or  favorable.  Most  weapon  system  programs,  regardless  of 
their  degree  of  complexity,  face  a  certain  amount  of  risk... 
unforeseen  and  possibly  unfavorable. 

The  concept  of  risk  has  often  been  used  to  denote  the 
probability  of  an  event  (4:11-2).  In  the  acquisition  envlr- 
nment,  cost-risk  is  the  probability  of  achieving  some  de¬ 
fined  program  event,  in  this  case,  a  cost  outcome.  If  the 
cost  exceeds  the  anticipated  outcome  a  cost  overrun  results. 
Conversely,  a  cost  underrun  is  any  program  cost  which  is 
less  than  the  anticipated  outcome. 

The  consequences  of  cost  overruns  are  well  recognized. 
The  affected  program  may  either  be  cancelled,  delayed  or 
funded.  Cancellation  results  in  unanswered  defense  needs, 
while  delays  result  in  Inefficient  program  execution  -  often 
at  a  higher  final  cost.  Funding  a  financially-troubled 
program  in  order  to  avoid  these  consequences  may  come  at  the 
expense  of  other  related  defense  programs. 

Beverly,  and  others,  state  that  cost-risk  has  commonly 
been  attributed  to  three  areas;  technical  risk,  schedule 
risk  and  estimating  risk,  Technical  risks  arise  from 
striving  to  achieve  maximum  performance  and  technological 
superiority.  The  'state-of-the-art'  is  often  advanced  be¬ 
yond  its  current  boundaries.  Schedule  risk  hinges  on  the 
ability  of  the  contractor  to  meet  contractual  delivery  terms 
with  a  product  of  acceptable  quality.  Estimating  risk  com¬ 
monly  results  from  vague  early  system  definition,  lack  of 
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sufficient  historical  data,  or  inherent  fluctuations  in  the 
prices  of  labor  and  materials.  While  all  three  risk  areas 
impact  cost,  their  effect  may  be  somewhat  complimentary 
(2:265).  For  example,  schedule  delays  may  stem  from  techni¬ 
cal  failures.  They  may  also  result  when  a  program  cannot 
continue  due  to  a  lack  of  funds  caused  by  an  inadequate  cost 
estimate.  Because  of  their  direct  and  complimentary  effects 
on  cost,  this  research  uses  cost-risk  in  the  context  of  an 
aggregate  measure  of  technical,  schedule  and  estimating 
risk.  A  program  cost  goal  will  be  viewed  as  that  program 
cost  which  balances  an  accepted  level  of  technical,  schedule 
and  estimating  risks, 

Definitions.  The  application  of  risk  assessment  to 
defense  weapon  system  acquisition  is  relatively  new;  there¬ 
fore,  widespread  agreement  on  common  terms  and  definitions 
has  not  yet  emerged  (14:35-39).  One  conclusion  from  a  1981 
risk  symposium  was  the  need  for  basic  definitions  and  common 
classifications  (15:175,2),  The  solidification  of  termi¬ 
nology  should  result  as  the  risk  assessment  discipline  ma¬ 
tures  . 

Rowe  and  Somers  categorize  risk  analysis  as  diagrammed 
in  Fig.  1.1.  They  view  risk  assessment  as  the  estimation  of 
the  risks  associated  with  given  program  alternatives,  while 
risk  management  is  the  action  taken  to  reduce  these  risks. 
Risk  analysis  is  considered  the  combination  of  risk  assess¬ 
ment  and  risk  management  (13:10). 
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1  Risk  Analysis  | 


Fig,  1,1  Risk  Analysis  Taxonomy  Adapted  from  (4:1—3) 

This  paper  accepts  that  taxonomy,  and  will  focus  solely  on 
risk  assessment.  For  further  clsrif ication ,  this  paper  uses 
a  modified  Defense  Systems  Management  College  (DSMC)  defini¬ 
tion  of  cost-risk  assessment,  "the  mathematical  probability 
of  achieving  or  not  achieving  acquisition  cost.,,  goals" 
(4:B-5),  Identifying  the  magnitude  of  the  deviation  from 
the  cost  goal,  along  with  the  deviation's  probability  of 
occurrence,  is  an  implicit  part  of  the  risk  assessment 
process.  Implied  in  this  definition  is  Worm's  concept  of 
reasonably  efficient  and  economical  practices  in  the  con¬ 
tractor's  and  government's  operation  (20:1), 

Specific  Problem  Statement 

While  program  cost  estimates  are  traditionally  pre¬ 
sented  a a  single,  unique  value,  the  probability  of  a- 
chievlng  that  cost  has  generally  been  ignored.  Not  until 
recently  has  there  been  an  emphasis  on  quantifying  cost- 
risk,  yet  risk  assessment  has  potential  benefits  when  the 
magnitude  of  unfavorable  consequences  is  significant. 


I - 1 

1  Risk  Assessment  ' 

- 

|  Risk  Management  T 
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Effective  cost-risk  assessment  identifies  and  helps  manage¬ 
ment  focus  attention  on  potential  cost  problems.  The  DoD 
manager  may  then  make  better  resource  allocation  decisions 
in  the  face  of  uncertainty  by  a<  'lying  risk  management  to 
appropriate  program  areas.  Also,  budget  decisions  made  with 
cost-risk  as  an  additional  consideration  should  reflect  more 
accurate  program  cost  goals,  Objective  identification  of 
adequate  contingency  funding  will  also  result.  Request  for 
these  contingencies  during  the  budget  process,  a  concept 
that  the  Army  is  using  <  8 : 2 ) ,  may  further  DoD  long-run 
credibility  by  reducing  the  occurrence  of  cost  overruns. 

In  order  to  make  better  decisions,  acquisition  person¬ 
nel  need  to  know  the  probability  of  deviating  from  the 
program's  cost  goal  or  the  probability . of  an  outcome  falling 
within  a  specified  range  of  values.  The  following 
information  is  needed  to  determine  this  probibllity. 

1 ,  The  shape  of  the  total  cost  distribution  is 
the  underlying  statistical  distribution  for  the  cost  of  an 
entire  system.  This  distribution  describes  the  relative 
likelihood  of  each  possible  cost  outcome  for  the  system. 

2 ,  The  measure  o f  location  describes  where  the 
mean,  median  or  mode  are  located  on  the  cost  distribution. 
The  mode,  or  most  likely  cost  outcome  is  typicully  the  point 
cost  estimate  for  the  program  ( 16:  130) . 

3  .  The  measure  o 1  s c a  1  e  indicates  the  amount  of 
variance  in  the  distribution,  A  large  variance  indicates  a 
large  dispersion  around  the  mean  and,  therefore,  a  greater 


potential  program  cost-risk.  A  good  variance  estimate  is 
crucial  in  cost-risk  assessment.  While  items  1  and  2  are 


necessary  information  for  cost-risk  assessments,  they  do  not 
impact  the  thrust  of  this  research. 

While  this  information  is  sufficient  for  identifying 
cost-risk,  it  is  seldom  availab3,e  for  the  entire  system. 
Current  cost-risk  techniques  divide  the  system  into  compo¬ 
nents,  or  identifiable  activities,  which  collectively  in¬ 
clude  all  required  program  tasks.  System  components  are 
typically  Work  Breakdown  Structure  (WBS)  elements;  however, 
Contract  Line  Items  or  other  categorizations  may  be  used 
(16:129).  The  cost  of  each  system  compohent  is  estimated 
separately.  Because  the  shape  of  the  cost  distribution  for 
each  system  component  is  rarely,  if  ever,  known,  analysts 
make  distributional  assumptions.  The  Normal,  Beta  and  Tri¬ 
angular  distributions  are  commonly  used  (18:195).  The  var¬ 
iance  of  each  component  is  determined  by  estimating  the  most 
optimistic,  most  pessimistic  and  most-likely  cost  outcomes. 
The  component  variances  are  then  added  together  to  yield  a 
total  system  cost  variance.  This  allows  cost-risk  assess¬ 
ments  to  be  made  around  the  total  system  cost  estimate.  This 
process  will  be  discussed  more  in  Chupter  2. 

It  is  the  authors'  opinion  that  the  methods  currently 
used  to  assess  total  system  cost-risk  are  inadequate.  Al¬ 
though  cost-risk  assessment  is  inherently  subjective,  cost- 
risk  is  also  mis -estimated  because  of  invalid  assumptions 
about  the  relationships  between  system  components.  Most 
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often,  components  are  assumed  to  be  completely  Independent 
of  each  other;  that  is,  cost  performance  in  each  system 
component  is  not  associated  with  the  cost  outcomes  of  other 
system  components.  In  other  words,  an  increase  In  the  cost 
of  one  of  the  system  components  does  not  imply  an  increase 
or  decrease  in  the  cost  of  any  other  system  component, 
Assuming  component  independence,  the  cost  variance  for  the 
total  system  is  simply  the  sum  of  the  component  variances. 
This  simplifying  assumption  is  unrealistic  because  it  under¬ 
states  the  system  cost  variance  whenever  its  component  coBts 
have  a  positive  relationship  to  each  other. 

The  counter-assumption  to  independence  is  the  assump¬ 
tion  that  there  is  maximum  positive  linear  dependency  among 
all  system  components.  With  this  assumption,  the  cost  var¬ 
iance  for  the  total  system  can  also  be  expressed  entirely  in 
terms  of  individual  component  variances,  Under  this  assump¬ 
tion,  an  Increase  in  the  cost  of  a  system  component  would 
cause  a  totally  predictable  increese  in  all  of  the  other 
system  components.  However,  this  approach  overstates  the 
system  cost  variance  by  assuming  that  the  strength  of  a 
component's  cost  influence  on  all  other  components  is  so 
overwhelming  as  to  be  totally  predictable.  The  overstate¬ 
ment  of  risk  occurs  because  cost  changes  are  assumed  to 
affect  components  that,  in  reality  would  not  be  affected, 
could  be  negatively  affected,  or  have  only  a  weak  positive 
affect.  For  this  reason,  assuming  maximum  positive  linear 
dependency  is  unrealistic.  This  will  be  addressed  further 
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in  Chapter  3. 
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Intuitively,  cost-risk  dependence  among  some  system 
component  costs  exists,  but  the  strength  of  the  relation¬ 
ships  will  vary  between  component  pairings  and  should  seldom 
be  strong  enough  to  correspond  to  maximum  dependency.  Sta¬ 
tistical  theory  exists  for  identifying  total  system  variance 
under  these  conditions.  The  details  will  be  discussed  in 
Chapter  3.  For  now,  suffice  it  to  say  that  the  total  vari¬ 
ance  io  a  function  of  the  individual  variances  and  a  covar¬ 
iance  term  for  each  component  pair.  Wherever  dependency 
exists  between  component  pairs,  the  covariance  indicates  the 
direction  of  dependence:  positive  or  negative,  A  positive 
covariance  between  two  components  indicates  that  a  rise  in 
the  cost  of  one  component  will  be  reflected  by  a  rise  in  the 
cost  of  the  second  component.  Slmilarily,  a  decrease  in  the 
cost  of  one  component  will  be  reflected  by  a  decrease  in  the 
cost  of  the  second  component.  On  the  other  hand,  a  negative 
covariance  indicates  that  component  costs  will  tend  to  move 
in  opposite  directions.  The  strength  of  the  relationship 
can  be  determined  after  a  simple  scaling  procedure. 

Current  risk  assessment  techniques  do  not  require  an 
estimate  of  the  covariance  among  system  components,  As 
such,  the  information  that  is  currently  used  is  insufficient 
to  accurately  measure  total  system  cost  variance  for  a 
program  with  dependent  components.  The  resulting  system 
cost-risk  is  inherently  distorted. 
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Research  Objective 


The  purpose  of  this  research  is  to  develop  a  methodolo¬ 
gy  which  produces  a  more  realistic  evaluation  of  cost-risk. 
Specifically,  the  goal  is  to  estimate  the  total  system  cost 
variance  given  the  existence  of  component  dependencies, 

Current  statistical  theory  provides  a  solution  to  the 
problem.  However,  the  solution  requires  reasonable  esti¬ 
mates  of  the  covariances  between  component  pairs.  Histori¬ 
cal  databases  generally  provide  an  inadequate  basis  for 
estimating  these  values,  which  means  that  program  management 
personnel  or  knowledgeable  system  experts  must  be  called 
upon  to  provide  subjective  measures  of  dependency.  The 
problem  is  that  these  experts  seldom,  if  ever,  think  about 
dependencies  in  terms  of  covariances.  It  would  be  a  gross 
understatement  to  say  that  most  experts  would  feel  uncom¬ 
fortable  using  covariance  to  describe  dependence  within  a 
program.  The  key  to  solving  this  problem  is  to  soliclte 
information  in  terms  that  relate  to  the  experts  frame  of 
reference.  Failure  to  do  so  may  result  in  receiving  no 
information  or  information  which  provides  a  distorted  pic¬ 
ture  of  the  expert's  true  opinion. 

The  methodology  developed  in  this  research  must  be  able 
to  convert  this  information  into  covariance  estimates  that 
capture  the  component  dependencies  implied  in  the  expert's 
responses.  Obviously,  the  adequacy  of  the  conversion  pro¬ 
cess  is  critical  if  the  total  system  cost  variance  is  to 
adequately  reflect  the  'real'  cost-risk. 


Development  of  this  methodology,  therefore,  requires 


that  the  following  specific  research  objectives  be  met! 

1.  Develop  a  methodology  to  translate  subjective 
inputs  into  a  measure  of  component  dependency 
with  a  logical  and  rational  basis. 

2.  Convert  the  component  dependency  measurement 
and  other  related  cost  data  into  a  range- 
stated  estimate. 

3.  Validate  the  predictive  abilities  of  this 
technique . 

The  methodology  to  meet  these  objectives  will  be  detailed  in 
Chapter  3. 
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Methodologies  for  quantifying  program  cost-risk  within 
DoD  aro  still  evolving.  Although  these  methodologies  differ 
somewhat  from  each  other,  they  follow  a  common  risk  assess¬ 
ment  approach.  The  purpose  of  this  chapter  is  to  describe 
this  general  risk  assessment  process  through  a  review  of  the 
current  literature. 


Scope  and  Limitations 

This  chapter  provides  an  overview  of  cost-risk  assess¬ 
ment.  It  does  not  intend  to  provide  a  how-to  procedure  nor 
on  in-depth  explanation  of  any  one  risk  assessment  tech¬ 
nique.  Instead,  it  summarizes  only  the  general  concept 
behind  risk  assessment,  with  special  interest  on  the  more 
limited  problem  of  determining  the  total  system  cost  vari¬ 
ance.  As  mentioned  in  Chapter  1,  determining  the  shape  and 
measure  of  location  of  the  total  cost  distribution  are 
beyond  the  scope  of  this  research.  This  paper  does  assume 
that  all  component  cost  estimates  (Including  the  low  and 
high  estimates,  which  will  be  discussed  later)  are  avail¬ 
able. 

Professional  journals  were  included  in  the  literature 
search,  but  they  are  oriented  toward  risk  assessment  in 
commerce  and  personal  investment.  Their  contribution  to 
this  specific  risk  assessment  problem  was  not  significant. 


For  this  reason,  concepts  in  this  chapter  are  primarily 
documented  in  government  studies,  articles  from  symposium 
proceedings,  and  official  literature  from  the  cost  analysis 
and  program  management  communities  of  the  three  military 
services . 

Organization 

Following  a  brief  background  of  risk  analysis,  this 
chapter  will  review  the  general  risk  assessment  approach. 

It  discusses  the  techniques  used  to  determine  distributional 
shape,  measure  of  location,  and  variance  for  the  components 
and  how  these  are  transformed  into  the  equivalent  lnforma-* 
tion  for  the  total  system.  It  concludes  with  a  portrayal  of 
this  total  system  information  in  a  cost-probability  rela¬ 
tionship. 


Background 

The  history  of  DoD  risk  assessment  is  quite  recent. 

The  post-Korean  conflict  era  found  the  military  services 
with  ample  funding  for  weapon  system  development  and  acqui¬ 
sition.  Cost  growth  was  generally  tolerated  and  easily 
absorbed  by  the  national  budget.  Events  changed  in  the 
1960's  and  1970's  as  the  Federal  budget  expanded  in  social 
entitlements.  As  competition  for  scarce  Federal  resources 
intensified,  military  cost  growth  quickly  became  more  polit¬ 
ically  sensitive.  This  scrutiny  of  DoD  management  fathered 
the  need  to  apply  risk  concepts  in  order  to  identify  and 
control  acquisition  costs. 
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In  the  wake  of  this  period  of  cost  growth  sensitivity, 
the  1981  Defense  Acquisition  Improvement  Program  highlighted 
DoD's  cost-risk  awareness.  Mr  Carlucci's  Initiative  #11 
'recommended'  an  increased  effort  to  quantify  risk  within 
the  DoD,  and  directed  the  military  services  to  adopt  a 
method  to  budget  funds  for  risk  and  uncertainty  (10:55). 

Early  work  with  DoD  risk  assessment  was  done  primarily 
by  the  RAND  Corporation.  Eventually,  the  problem  received 
increased,  cooperative  attention  from  industry,  government 
and  academe.  The  first  symposium  to  include  the  management 
of  risk  in  the  defense  acquisition  process  was  hosted  by  the 
University  of  Southern  California  (U.S.C.)  in  1979.  A  sec- 

i’ 

ond  workshop,  held  at  the  Air  Force  Academy  in  1981,  was  co¬ 
sponsored  by  U.S.C.  and  the  Air  Force  Business  Research 
Management  Center.  The  most  recent  workshop  met  at  the  DSMC 
in  1983  (13:6).  The  workshop  has  now  evolved  into  a  biannual 
Defense  activity. 

Despite  this  increased  emphasis  on  risk  applications, 
most  current  DoD  acquisition  policies  do  not  require  formal, 
quantitative  risk  assessments.  One  exception  is  the  Army's 
Total.  Risk  Assessing  Cost  Estimate  (TRACE)  program  which 
requires  risk  assessments  fur  selected  programs  and  incorpo¬ 
rates  the  results  into  the  budget  process  (10:61). 

The  Risk  Assessment  Process 

Cost-risk  assessment  techniques  follow  the  general 
approach  illustrated  in  Fig.  2.1.  This  process  has  three 


basic  steps  (21:3-4). 


Fig,  2.1,  Risk  Assessment  Process  Adapted  from  (18:194) 


First,  the  weapon  system  is  broken  into  its  components  at 
some  specified  level  of  detail.  The  breakdown  of  each 
system  should  go  as  low  as  necessary  to  include  those  compo¬ 
nents  which  are  considered  to  have  particularly  high  risk 
(11:250),  The  cost  uncertainty  for  each  component  is  then 
represented  by  a  distribution  of  the  possible  costs  for  the 
component.  Second,  the  component  distributions  are  trans¬ 
formed  into  a  cost  distribution  for  the  total  system  using 
some  aggregation  method  in  order  to  reflect  the  amount  of 
uncertainty  in  the  total  system  cost.  Third,  this  total 
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distribution  ia  expressed  in  probabclistic  terms  to  aid 
program  management  decisions.  The  following  sections  de¬ 
scribe  these  steps  in  more  detail. 

Step  One .  Component  costs  are  point  estimates  that  ere 
derived  from  cost  estimating  techniques  such  ass  parametric 
(cost  estimating  relationships),  detailed  (bottoms-up) , 
analogy,  and  constant  multiplier  (factor)  ( D s 136).  These 
techniques  require  information  about  component  physical  and 
performance  characteristics,  manhour  and  material  require¬ 
ments,  or  a  subjective  estimate  about  the  similarities  of 
the  component  to  other  components  for  which  the  costs  are 


known . 


f 
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The  potential  cost  variability  around  these  point  esti¬ 
mates  is  often  represented  by  a  probability  density  function 
(p.d.f.).  The  shape  of  the  p.d.f.  often  has  the  following 
characteristics : 

1.  fixed,  positive  upper  and  lower  bounds 

2.  not  necessarily  symmetric 

3.  unimodal 

4.  computationally  simple  (18:195) 

Flexibility  is  also  desirable,  such  that  changing  the  values 
of  the  distribution's  parameters  allows  it  to  nssume  many 
variations  of  its  general  shape.  This  characteristic  offers 
adaptability  in  representing  a  variety  of  cost  patterns 
(21:5).  Most  analysts  use  the  Beta  distribution.  Others 
prefer  the  Gamma,  Normal  or  Weibul  distributions  because 
these  are  un-bounded  in  at  least  one  direction,  and  there- 
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fore,  leave  open  the  possibility  of  unusually  high  and/or 
low  cost  outcomes  (18:195).  The  unbound  portion(s)  of  the 
curve  run  asymptotic  to  the  x-axis;  therefore,  the  proba¬ 
bility  of  extremely  high  and/or  low  values  approaches  zero 
(21:5),  The  triangular  distribution  may  also  be  appropriate 
(18:195).  The  selection  of  a  distribution  shape  is  largely 
a  matter  of  analyst  preference  and  insight  into  the  possible 
cost  outcomes.  Two  or  more  distribution  shapes  may  be 
reasonable  choices  in  many  situations. 

Once  a  distribution  has  been  selected,  the  program 
management  team  often  needs  to  estimate  only  three  values  to 
determine  the  distribution’s  variance:  low  cost,  most- 
likely  cost  and  high  coat.  The  low  estimate  should  be  that 
cost  which  results  from  the  most  optimistic  conditions.  The 
probability  of  achieving  a  cost  below  this  value  is  zero  for 
a  bounded  distribution.  The  most-likely  estimate  is  the 
most-probable  cost,  or  the  mode  of  the  distribution.  The 
high  estimate  should  be  that  pessimistic  cost  which  reflects 
the  worst  conditions.  The  probability  of  exceeding  this 
high  value  is  also  zero  for  a  bounded  distribution 
(17: A-50) . 

Given  these  three  values,  a  formula  may  then  be  used  to 
calculate  the  variance.  For  example,  the  approximate  vari¬ 
ance  of  a  Beta  distribution  is  calculated  by  squaring  one- 
sixth  of  the  difference  between  the  high  and  low  estimates 
(9:138}  12:30). 

Naturally,  the  low,  most-likely,  and  high  estimates  are 


subjective.  Kazanowski  feels  that  the  most-likely  value  is 
generally  the  most  difficult  to  estimate  because  of  estima¬ 
tor  indifference  to  a  broad  mid-range  of  values  (9:138). 

For  certain  distributions  this  is  true.  Fig.  2.2(e)  repre¬ 
sents  such  a  case  where  the  estimator  may  be  reasonably 
certain  about  the  upper  and  lower  bounds,  but  a  large  degree 
of  indifference  exists  for  the  most-likely  value. 


(b) 


Fig.  2.2.  Coat  Distribution  Examples 


However,  the  distributional  shapes  shown  in  Flg.s  2.2(b)  and 
2.2(c)  are  so  greatly  skewed  in  one  direction  that  the  high 
or  low  cost  may  be  the  most  difficult  to  estimate. 

Analysts  may  feel  uncomfortable  in  estimating  absolute 
low  or  high  costs,  particularly  if  they  perceive  extreme 
skewness  and  have  selected  an  unbounded  distributional 
shape.  In  those  cases,  the  low  and  high  estimates  need  not 
be  absolutes.  Instead,  the  analyst  may  select  a  reasonably 
low  cost  and  estimate  the  associated  probability  of  under- 
running  it.  He  also  selects  a  reasonably  high  cost  and 
estimates  the  probability  of  overrunning  it.  For  example, 


McNichols  provides  a  revised  formula  for  calculating  the 
approximate  variance  for  a  Beta  distribution  when  the  5  and 
95  percentile  costs  are  given.  The  formula  is  the  differ¬ 
ence  between  the  high  (x  and  low  (x  Q5)  estimates  di¬ 
vided  by  3,2,  quantity  squared  (12t36),  Similarly,  with  a 
triangular  distribution,  Wilder  and  Black  provide  a  means  of 
calculating  the  absolute  high  and  low  costs  using  any  per¬ 
centile  ( 1 7  *  A — 50 ) .  Once  the  absolute  high  and  low  costs 
have  been  calculated,  they  may  be  used  in  a  formula  to  find 
the  variance  for  the  distribution. 

Step  Two .  The  next  step  in  the  process  is  to  transform 
the  cost  estimate,  distribution  shape  and  the  variance  for 
each  component  into  an  overall  cost  distribution  for  the 
total  weapon  system  (21:4),  A  common  technique  for  deter¬ 
mining  the  shape  of  the  distribution  is  the  method  of  mo¬ 
ments,  which  was  first  applied  by  McNichols  in  1976 
(18:195A),  The  technique  is  generally  used  in  conjunction 
with  parametric  cost  estimating  methods.  As  a  result,  it 
generally  relies  on  a  large  historical  database  to  determine 
the  probability  distributions  for  the  components  and  the 
total  system  (12i20). 

The  method-of-moments  characterizes  each  component  cost 
distribution  by  four  additive  moments.  The  first  additive 
moment  is  the  mean  of  the  distribution,  that  is,  the  first 
moment  about  the  origin.  The  second  and  third  additive 
moments  are  central  moments  (moments  taken  about  the  mean). 
The  second  moment  is  the  variance,  which  measures  squared 


deviations  about  the  mean,  while  the  third  moment  is  used  to 
measure  asymmetry.  The  fourth  additive  moment  measures 
peakedness,  and  is  formulated  using  the  second  and  fourth 
central  moments  ( 1 6 s 130) .  The  four  additive  moments  are  ex* 
pressed  analytically  so  that  the  formulas  have  additive 
properties.  For  example,  the  second  moment  (variance)  for 
the  total  system  is  the  summation  of  ell  the  component 
second  moments.  Therefore,  the  shape  of  the  total  coat 
distribution  for  a  system  with  independent  components  is 
found  by  summing  the  similar  moments  of  every  component. 

The  analyst  may  then  'fit'  a  selected  distribution  to  the 
four  moments  with  the  aid  of  computerized  routines  (lf>:130). 

The  measure  of  location,  or  the  point  eetlmate,  for  the 
total  system  cost  has  not  been  calculated  in  a  wholly  con¬ 
sistent  manner.  The  method-of-moments  uses  the  sum  of  the 
component  first  moments  (means)  as  the  total  system  cost 
estimate  ( 4 1 F— 2 5 1  21 1 36 ) .  Kaaanowski,  who  used  a  less  so¬ 
phisticated  approach  to  risk  assessment  than  the  method-of- 
moments,  also  summed  the  component  means.  He  calculated 
individual  component  means  with  e  formula  using  low,  most- 
likely,  and  high  costs,  similar  to  the  method  discussed 
earlier  for  calculating  component  variances  (9:15b).  In 
contrast,  the  Army's  TRACE  model  uses  the  sum  of  the  point 
estimates  (modes)  as  the  total  system  cost  (8:46).  Another 
method  is  used  by  the  Air  Force's  computerized  RISK  model, 
where  the  median  value  of  the  total  system  cost  distribution 


median  is  that  value  whose  cumulative  probability  of 
occurrence  is  50%. 

The  literature  is  significantly  more  consistent  on 
calculating  the  total  system  cost  variance  than  it  is  on  the 

,  i 

measure  of  location.  With  the  few  exceptions  that  will  be 

discussed  shortly,  analysts  assume  independence  between 

components  and  sum  their  variances  for  the  total  system  cost 

variance  ( 16: 1 30 s  21:39j  12:20).  The  method-o£-mements 

typifies  this  approach,  where  the  variance  of  the  total 

system  is  the  sum  of  each  component  second  moment  (16:130). 

♦ 

The  literature  clearly  documents  the  need  to  include  covar¬ 
iance  terms  in  order  to  compute  total  system  cost  variance 
when  components  ore  dependent  (21:39:  ,9:153). 

Step  Three .  The  final  step  translates  the  total  system 
cost  distribution  into  a  cost-probability  relationship. 

While  the  probability  density  function  provides  the  decision 
maker  with  a  graphic  picture  of  the  cost  variability,  prob¬ 
ability  assessments  are  difficult  in  this  format  because 
probabilities  are  represented  by  the  area  under  the  curve 
(4:11-8).  A  cumulative  distribution  function  (c.d.f.) 
translates  the  area  under  the  curve  for  each  cost  value  into 
a  cumulative  probability.  The  c.d.f. ’s  in  Fig.  2.3  allow 
the  program  manager,  or  other  decision  maker,  to  assess  the 
probability  of  over  or  underrunning  a  given  program  cost. 

For  example,  according  to  Fig.  2.3(a),  there  is  an  80% 
chance  that  a  hypothetical  program  will  cost  less  than  $200. 


23 


(a)  (b) 

Fig.  2.3  Cumulative  Coat  Distributiona 

Stated  another  way,  the  probability  of  overrunning  $200  ia 
20%.  Cumulative  diatribution  functlona  also  reveal  the 
probability  of  a  program  outcome  between  any  two  coats, 
Figure  2.3(b),  for  example,  ahowa  the  probability  of  a- 
chievlng  a  coat  between  $100  and  $200  la  80%  (90%-10%).  The 
probability  of  a  aimilar  coat  outcome  on  Figure  2.3(a)  ia 
only  60%.  A  steeper  c.d.f.,  therefore,  indicates  a  smaller 
cost  variance  and  less  program  coat-risk. 

Current  Peraoectives  on  Component  Dependency 

Moat  current  risk  assessment  techniques,  such  aa  the 
method-of-mornents ,  rely  on  the  erroneous  assumption  of  inde¬ 
pendent  system  components.  The  problem  with  this  assumption 
is  that,  in  most  circumstances ,  it  understates  true  program 
cost-risk  ( 9 1 1 5 0 ,  16*1 30 ) . 

The  literature  explains  why  analysts  continue  to  rely 
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on  this  assumption  in  spite  of  its  weakness.  Black  calLs 
the  independence  assumption  "troublesome"  (16:130),  but  cites 
computational  ease  as  one  reason  for  its  use.  Analysts  have 
not  been  able  to  apply  the  method-of -moments ,  for  example, 
to  situations  with  realistic  component  dependency.  Because 
the  method-of-moments  technique  is  mathematically  rigorous, 


derivation  of  moment  functions  with  additive  properties  is 
not  possible  unless  component  dependencies  are  perfectly 
predictible  (16:133).  The  technique  becomes  unwieldy  if  it 
incorporates  any  deviations  from  a  strict  linear  relation¬ 
ship  between  components. 

Worm  feels  that  the  continued  reliance  on  the  inde¬ 
pendence  assumption  rests  in  the  difficulty  in  expressing 
dependency  between  components  in  a  meaningful  way.  Quanti¬ 
fying  covariance  terms  has  been  elusive  because  of  the  lack 
of  historical  data  or  of  a  method  to  estimate  covariance 
from  subjective  information  (21:39). 

Only  limited  offorts  hove  been  taken  to  resolve  the 
dependency  problem.  Wilder  and  Black's  approach  was  to 
place  an  upward  bound  on  the  cost-risk  of  component  depend¬ 
ency  rather  than  estimate  its  true  affect.  Two  situations 
were  assessed  -  independence  and  complete  linear  dependence, 


If  we  muke  the  opposite  assumption,  i.e.,  that 
there  is  complete  [positive]  linear  dependence 
among  the  project  elements,  in  effect,  we  say  that 
any  problem  with  any  element  will  be  reflected 
in  all  elements,  and  conversely  any  'good  luck' 
will  be  similarly  reflected.  This  assumption  may 
not  be  valid  in  many  situations,  but  we  feel  that 
it  is  closer  to  reality  than  the  independency 
assumption,  and  the  region  between  the  two 
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assumptions  may  be  considered  to  bound  the  set  of 
Intermediate  outcomes  [16:130]. 
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The  method-of-moment s  was  used  to  develop  a  cumulative 
cost  distribution  for  both  assumptions.  The  two  curves,  as 
shown  in  Fig,  2.4,  bracket  the  actual,  unknown  curve.  To 
apply  this  concept,  Wilder  and  Black  derived  a  series  of 
dependent  moments.  The  first  dependent  moment  is  the  mean 

tih 

of  the  distribution,  while  the  remaining  k  dependent  mo- 

t*  h  t  h 

ments  are  the  ktn  root  of  the  kc  central  moment,  The 
relationship  between  the  dependent  end  the  independent  mo¬ 
ments  is  shown  below: 


(2.1) 

(2.2) 


C4-3C2‘ 


(2.3) 

(2.4) 

(2.5) 


where 


k  h  dependent  additive  moment 
k^.  independent  additive  moment 
k£.  central  moment 
first  origin  moment 


The  dependent  moments  have  the  same  additive  properties  as 
the  Independent  moments.  Therefore,  the  dependent  moments 
of  the  total  program  ere  the  sums  of  the  dependent  moments 
of.  each  component  (16:130-131).  Referencing  Fig,  2,4,  the 
independent  curve  underestimates  the  program  risk  for  sys- 


terns  with  predominantly  positive  dependence,  while  the  de¬ 
pendent  curve  overestimates  it.  "It  is  a  matter  of  judgement 
to  determine  where  in  this  area  the  'truth*  lies"  (16:131). 


Fig.  2.4.  Cumulative  Distribution  Functions 

Adapted  from  (16:132) 

Worm  offers  a  second  approach.  Rather  than  bound  the 
problem,  he  attempts  to  structure  the  dependency  between 
components  .  Using  contract  pricing  items  for  system  compo¬ 
nents,  Worm  breaks  each  component  into  Independent  and  de¬ 
pendent  portions.  The  dependent  variation  in  each  component 
is  thought  to  stem  from  a  common  set  of  exogenous  factors 
which  affect  all  system  components.  To  illustrate  this 
concept,  two  components,  such  as  labor  and  material  costs, 
may  both  be  excessive  because  of  design  immaturity.  The 
dependency  between  these  components  is  reflected  by  the 
influence  of  the  common  factor,  design  maturity,  on  both 
components.  When  more  than  one  of  the  exogenous  factors  are 
present,  their  effect  on  each  component  is  estimated  cumula- 
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tiveiy.  The  contribution  of  all  exogenous  factors  is  summed 
for  each  component  and  the  aggregated  value  collectively 
becomes  a  single,  separate  and  independent  element.  Each 
component  is  then  viewed  as1  the  sum  of  two  independent 
random  variables  and  the  traditional  additive  properties  of 

t. 

the  method-of-moments  apply  (21:39-43). 

The  Authors 1  Viewpoint 

The  successful  inclusion  of  component  dependence  in 
cost-risk  assessment  has  been  elusive.  Two  approaches  to 
dependency  were  observed,  The  authors'  believe  that  neither 
approach  satisfactorily  resolves  this  problem. 

Wilder  and  Black  bounded  the  problem  by  taking  a  total¬ 
ly  dependent  and  independent  perspective.  Neither  assump¬ 
tion  is  valid.  They  both  intentionally  mis-state  true  pro¬ 
gram  risk.  Quantification  of  covariance  terms  was  unneces¬ 
sary  because  the  components  had  either  no  influence  on  each 
other'  or  were  maximum  linearly  dependent.  Components  with 
negative  dependence  were  not  considered.  While  some  insight 
was  guined,  its  information  value  is  limited  by  the  broad 
range  of  subjectivi ty  left  to  the  decision  maker. 

Worm's  approach  toward  dependency  is  also  inadequate. 
Even  if  all  the  exogenous  factors  are  correctly  identified, 
a  significant  weakness  remains.  By  structuring  dependency 
around  »  sot  of  external  influences,  the  interrelations!!! p 
of  the  components  themselves  are  ignored.  As  shown  in  Fig. 

2  .  '3  ( a  ) ,  Worm's  cost  estimating  relationships  do  not  account 
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for  Lhe  direct  association  between  system  components.  The 
true  cost  dependence  relationships  which  need  to  be  modeled 
are  shown  in  Fig.  2.5(b). 
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Fig.  2.5  Component  Cost  Dependence  Relationships 


Conclusion 

This  chapter  reviewed  the  general  risk-assessment  pro¬ 
cess.  Techniques  were  briefly  described  for  estimating  a  . 
component's  cost,  variance,  and  distributional  shape. 

Methods  were  described  to  transform  this  component  informa¬ 
tion  into  the  equivalent  Information  for  the  total  system. 
The  difficulties  in  incorporating  component  dependence  into 
u  total  system  cost  variance  were  discussed.  Two  approaches 
to  this  problem  were  reviewed,  along  with  their  limitations. 


Ill,  Methodolo 


The  literature  review  on  cost-risk  assessment,  sum¬ 
marised  in  Chapter  2,  revealed  that  cost  dependent  relation¬ 
ships  are  not  used  in  current  cost-risk  assessment  methods. 
This  research  attempts  to  overcome  that  limitation.  The 
chapter  is  divided  into  two  sections.  The  first  section 
develops  a  cost-risk  assessment  methdd  that  incorporates 
component  cost  dependence.  The  second  section  explains  the 
approach  used  to  demonstrate  the  significance  of  component 
dependence  in  cost-risk  assessment,  and  describes  the  pro¬ 
cedure  used  tq  verify  the  internal  validity  of  the  proposed 
method.  Fig.  3,1  shows  a  general  scheme  for  the  research. 
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Figure  3.1  Generalized  Cost-risk  Assessment  Method 


Cost- r isk  Assessment  Methodology 

Total  System  Cost  Variance .  In  the  generalized  form, 
total  system  cost  variance  can  be  expressed  as  the  sum  of 
component  variances  and  the  covariances  between  components. 
This  generalized  expression  is  shown  in  the  following  equation 


(3.1) 


n 


TOT  SYS  COST  VAR 


■£  £ 

i-lj-l 


Cov(Xi,Xj) 


where 


n  ■  number  o£  system  components 
.X.  ■  ith  component  of  the  system 
X^,Xj  m  component  pair  within  the  system 


and 

Cov(X1,Xj)  -  Var(Xi)  when  i-J 


(3.2) 


Winkler  and  Hayes  (19:186)  define  covariance  in  terms 
of  expected  values  as 

Cov<X1,XJ)  -  E  [  (XA  -  E(Xi))  ( X j  -  E(Xj))  ]  (3.3) 


When  dealing  with  population  samples*  the  expression  is 
modified  to 


Cov(X1,XJ)  -  E  [  <Xt  -  (Xj  -  Xj )  ]  (3.4) 


The  associative  property*  when  applied  to  expected  values, 
states  that  E(ab)  ■  E(ba)  .  The  order  in  which  the  terms 
are  multiplied  has  no  effect  on  the  expected  value.  There¬ 
fore,  Eq  (3.4)  can  be  written  as: 
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E  [  <X±  -  X±)  (Xj  -  Xj)  ] 

-  E  [  (X 


-  Xj)  (Xt  -  X±)  ]  (3.5) 
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Therefore , 


Cov(Xi , X j )  -  CovCXj,Xi) 


(3.6) 
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Tho  covariance  between  any  two  pair  of  components  appears 
twice  in  Eq  (3.1),  once  as  Cov(X^,Xj)  and  once  as 
Cov(XjlX1)*  By  substituting  CovCX^Xj)  for  CovCXj^) 
whenever  j  Is  greater  than  i,  the  total  number  of  covariance 
terms  in  Eq  (3.1)  is  reduced , by.  half .  £>nly  the  covariance 
terms  CcvCX^Xj)  for  i  less  than  j  remain  in  the  equation. 
Furthermore,  each  term  ia  multiplied  by! two  to  account  for 

i 

the  substitution, 

In  the  cost-risk  assessment  process,  for  a  given  system 
having  two  components,  X^  and  X2 ,  the  determination  of  total 
system  cost  variance  requires  three  steps. 

Step  One .  Determine  the  variance  for  the  cost 
distribution  of  Component  Xj  and  Component  X2.  This  is 
usually  accomplished  by  determining  the  low,  most-likely, 
and  high  cost  values  and  applying  a  variance  assessment 
technique  as  described  in  Chapter  2. 

Step  Two .  Determine  the  covariance  between  compo¬ 
nents  X ^  and  X2.  Currently,  this  is  a  major  problem  in 
acquisition  program  cost  estimation.  A  latter  part  of  this 
chapter  will  offer  a  subjectively-based  means  of  measuring 
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the  strength  of  a  dependency  relationship. 

Step.  Three .  The  cost  variances  and  covariances  of 
each  component  and  its  pairwise  influence  are  summed.  For 
the  two  component  system,  the  following  terms  are  generated: 
VarCXj),  VarCXj),  CovCXj.Xj),  Cov(X2,Xj).  From  Eq  (3.6), 
the  third  and  fourth  terms  are  equal,  so 


Cov(XltX2)  +  Cov(X2,X1)  -  2Cov(XltX2) 


(3.7) 
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The  total  system  cost  variance  can  then  be  expanded  in 
equation  form  from  Eq  (3.1)  to 
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TOT  SYS  COST  VAR  -  Var(Xj)  +  Var(X2) 

+  2Cov(X1,X2) 


(3.8) 
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Limiting  Combinations  and  Covariance  Terms .  In  a  sys¬ 
tem  with  more  than  two  components,  more  than  two  covariance 
terms  must  be  expressed,  one  for  each  pairwise  combination. 
T1  ;  number  of  covariance  terms  required  is  the  combination 
expression : 


0)*n 


I  /  2  !  (n-2) ! 


(3.9) 


where 


n  ■  number  of  components  in  the  system 


The  number  of  pairwise  combinations  rises  quickly  as  the 
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number  of  components  In  the  system  rises.  For  example,  in  a 
system  with  three  components,  the  number  of  covariance  terms 
required  is: 
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3!  /  2  ! (3-2) I  -  3 


(3.10) 


However,  the  required  combinations  for  a  system  with  ten 


components  1st 


CD- 


10 1  /  2 ! ( 10-2) !  -  45 


(3.11) 


Since  much  larger  systems  are  not  uncommon,  the  authors  will 


simplify  the  research  by  defining  a  system  as  a  linear 


series  of  components.  Components  will  have,  at  most,  only 
one  preceding  and  one  subsequent  task.  A  major  assumption 
with  this  approach  is  that  all  component  dependence  is 


captured  in  a  sequential  manner.  For  example,  dependence 


between  components  Xj  and  is  assumed  to  be  accounted  for 
by  the  intervening  component,  X2.  Component  X^  has  no  cost 
relation  to  Component  X3,  except  as  Indirectly  transferred 
through  Component  X2«  The  system,  therefore,  is  greatly 
simplified  because  it  has  only  * n— 1  *  covariance  terms  rather 


than  the  theoretical  maximum  in  Eq  (3.9). 


Covariance  Interpretation ,  The  preceding  paragraphs 
have  discussed  the  definition  of  covariance,  its  properties, 


and  how  it  affects  the  total  system  cost  variance.  While  a 


covariance  term  can  assess  the  association  between  two  com- 


8sL_ 


ponents,  it  is  not.  without  its  flaws.  There  are  three 
problems  with  covariance  -  all  caused  by  its  units  of  meas¬ 
urement  . 

First,  the  magnitude  of  the  covariance  term  is  extreme¬ 
ly  sensitive  to  the  components'  units  of  measurement.  If  the 
units  of  Component  X^  and  Component  X2  are  expressed  in 
dollars,  the  covariance  term  is  expressed  with  these  unitsi 
Cov(Xj,X2)  (dollars)  (dollars).  If,  however,  the  units  are 
expressed  in  other  terms,  such  as  cents,  the  magnitude  of 
the  covariance  will  change  even  though  the  relationship 
between  the  components  remains  the  same.  Essentially,  the 
magnitude  of  the  covariance  term  can  be  manipulated  by 
changing  the  units  of  the  dependent  components. 

The  second  issue  concerning  units  in  the  covariance 
term  is  that  the  units  do  not  make  sense.  If  Component 
and  Component  X2  have  units  of  pounds  end  miles  respective¬ 
ly,  the  resulting  covariance  term  will  be  expressed  in 
pound-miles.  This  expression  is  confusing  and  serves  no 
useful  purpose  in  the  determination  of  dependence  between 
the  two  components. 

The  third  problem  with  units  deals  with  the  compar¬ 
ability  of  different  covariance  terms.  The  different  units 
that  are  used  with  the  covariance  term  prohibit  the  compari¬ 
son  of  the  dependence  between  different  pairs  of  components. 
The  following  example  illustrates  this  problem. 

CovCX-^Xg)  is  expressed  in  the  same  units  used  to  measure 


in  pounds  and  Component  X2  is  measured  in  miles,  the  co- 
variance  between  the  components  is  measured  in  pounds  times 
miles.  Likewise,  if  Component  and  Component  X^  are 
measured  in  volts  and  square  feet,  respectively,  the  co- 
variance  between  them  is  measured  in  volts  times  square 
feet.  Obviously,  this  difference  in  units  of  measure  makes 
it  Impossible  to  compare  CovCX^Xj)  with  Cov(X3,X^).  It  is 
a  comparison  of  apples  and  oranges. 

Correlation  Coefficient  and  Variance-Covariance 
Relationships .  .  In  order  to  compare  covariances,  the  vari¬ 
ances  of  the  components  (component  costs  in  this  case)  are 
used  to  scale  the  covariance  torms.  The  outcome  is  the 
correlation  coefficient,  Rho  (R). 


xx  -  CovCX^Xj)  /  [  Var(Xt)  Var(Xj)  ]*  (3.12) 


i  j 


The  correlation  coefficient  ranges  from  -1  to  +  1,  A  value 
of  -1  corresponds  to  maximum  negative  linear  dependence;  a 
value  of  +1  corresponds  to  maximum  positive  linear  depend¬ 
ence;  and  a  value  of  0  corresponds  to  Independence  between 
the  components'  costs.  These  terms  were  defined  in 
Chapter  1, 

Eq  (3.12)  makes  it  possible  to  solve  the  problems  of 
units  of  measure  that  were  discussed  earlier.  For  Component 
and  Component  Xj  in  pounds  and  miles,  Eq  (3.12)  scales 
the  covariance  term  by  eliminating  the  units  of  measurement. 


(3.12)  is  modified  to  show  only  the  units  In  the  equation. 


R y  v  ■  (pound-miles)  /  [  ( pounds)^ (miles )^  (3.13) 

A1A2 

The  units  in  the  equation  cancel.  RhoY  Y  is  now  a  unitless 

*1*2 

value  whose  magnitude  has  been  scaled.  The  relationship 
between  Rho,  component  variances  and  component  covariances 
is  a  critical  relationship  for  this  research.  The  popular 
assumptions  of  component  independence  and  maximum  positive 
linear  dependence  are  based  on  this  relationship.  Later, 
this  relationship  will  be  used  as  a  major  part  of  the  cost- 
risk  assessment  method  that  this  research  proposes. 

The  Independence  Assumption .  In  Chapter  2  it  was  men¬ 
tioned  that  since  the  quantification  of  dependency  between 
the  cost  of  system  components  is  difficult  to  assess,  many 
cost-risk  assessment  methodologies  assume  component  inde¬ 
pendence.  This  assumption  allows  for  the  exclusion  of  the 
covariance  terms  from  the  total  system  cost  variance  equa¬ 
tion.  The  independence  assumption  means  that  there  is  no 
association  between  the  cost  of  one  component  and  the  cost 
of  any  other  system  component.  The  independence  assumption 
also  implies  a  Rho  value  of  aero  (19*187). 

Therefore , 

Rx  x  -  Cov(Xi,XJ)/[  Var(Xi)  Var(Xj)  ]*  -  0  (3.14) 

1  2 

If  Rho  is  zero,  the  numerator  of  the  ratio  must  be  zero. 
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Therefore,  the  independence  assumption  implies 


Cov(Xi,Xj)  -  0  (3.15) 

The  equation  for  total  system  cost  variance,  then,  can  be 
expressed  as 

TOT  SYS  COST  VAR  -  V,  Var(X4 )  (3.16) 

1*1  1 

The  effect  of  this  deletion  is  obvious.  Assessment  of 
covariance  terms  Is  unnecessary  under  the  independence 
assumption. 

The  Maximum  Positive  Linear  Dependence  Assumption. 

Since  the  assumption  of  total  independence  seems  to  violate 
our  intuitive  understanding  of  how  systems  are  developod, 
some  cost-risk  assessment  methodologies  assume  total  posi¬ 
tive  linear  dependence  between  component  costs.  Total  posi¬ 
tive  linear  dependence  means  that  if  the  cost  of  one  compo¬ 
nent  is  higher  than  expected,  not  only  will  the  the  cost  of 
all  subsequent  components  be  higher  then  expected,  but  the 
magnitude  of  the  overrun  can  be  predicted  with  absolute 
certainty.  In  essence,  the  total  system  cost  is  entirely 
determined  by  the  cost  of  the  first  component  in  the  system. 

This  assumption  also  enables  us  tc  compute  the  totil 
system  cost  variance  without  estimating  covariance  terms. 
Again,  the  expression  for  Rho  is  used  to  demonstrate  this 
phenomenon.  Total  positive  linear  dependence  between  two 


components'  costs  Is  represented  by  a  correlation  coeffi¬ 
cient  equal  to  +1.  When  Rho  is  one,  Eq  (3.12)  gives  the 
following  solution  for  the  covariance; 


Rx  x  -  Cov(XlfX2)/[  Var(X j )  Var(X2)  ]*  -  1  (3.17) 

12 


Cov(X1,X2)  -  [  Var(Xj)  Var(X2)  ]*  (3.18) 

In  general, 


Cov(XifXj)  «  R 


Var(Xi)  Var(Xj) 


(3.19) 


The  fact  that  Rho  falls  between  -1  and  +1  means  that  the 
Cov(Xj,X2)  is  at  its  maximum  value  when  Rho  is  one.  There¬ 
fore,  Eq  (3.1)  for  the  total  system  cost  variance  will  also 
be  maximized  when  total  positive  linear  dependence  is  as¬ 
sumed  , 

For  the  two  component  example,  making  the  substitution 
shown  in  Eq  (3.18)  in  the  aquation  for  total  system  cost 
variance  gives  the  following  expression; 


TOT  SYS  COST  VAR  -  Var(X1)  +  Var(X2) 

+  2  [  Var(X1)  Vur(X2)  ]*  (3.20) 


Under  the  assumption  of  total  positive  linear  dependence, 


total  system  cost  variance  can  be  expressed  without  the  use 
of  covariance  terms.  As  mentioned  In  previous  chapters, 
this  approach  Is  not  Intuitively  appealing  because  it  as¬ 
sumes  that  extreme  cost  outcomes  will  ripple  through  the 
entire  system.  Using  this  assumption,  therefore,  inflates 
the  total  system  cost  variance. 

The  Maximum  Negative  Linear  Dependence  Assumption. 
Component  cpst  dependencies  are  not  always  positive  -  a  low 
cost  in  one  component  does  not  always  mean  that  other  compo¬ 
nents'  costs  will  be  low;  a  high  cost  in  one  component  does 
not  always  mean  that  other  components'  costs  will  be  high, 
When  a  low  cost  in  one  compoiient  is  associated  with  a  high 
cost  in  another  component,  the  dependency  relationship  is 
negative.  The  assumption  of  maximum  negative  linear  de¬ 
pendency  sets  the  value  of  Rho  for  ell  pairwise  combinations 
of  components  equal  to  -1,  Iq  (3.19)  indicatss  that  if  Rho 
is  -1,  the  covariance  term  takes  on  its  smallest  possible 
value,  which  also  happens  to  be  negative. 

Cov(Xi,XJ)  -  -1  [  VanXt)  Var(Xj)  ]*  (3.21) 

Therefore,  Eq  (3.1)  will  be  minimized  when  total  negative 
linear  dependence  is  assumed. 

Conclusions  About  Total  Linear  Dependence .  Theeo  two 
assumptions,  total  positive  linear  dependence  and  total 
negative  linear  dependence,  generate  the  maximum  and  minimum 


Normally,  It  is  expected  that  some  covariance  terms  would  be 
positive  and  some  would  be  negative.  The  sum  of  the  co- 
variances  may  be  negative  or  positive,  depending  on  the  sign 
and  magnitude  of  the  Individual  terms.  To  the  extent  that 
this  sum  differs  from  aero,  the  total  system  cost  variance 
obtained  under  the  assumption  of  total  independence  will 
either  over-  or  underestimate  the  true  system  coat  variance. 

Estimating  the  Covariance.  As  Chapter  1. mentioned,  the 
database  does  not  exist  to  calculate  covariance  terms.  This 
necessitates  the  use  of  subjective  information  to  determine 
the  degree  of  cost  dependence  between  components.  This 
information,  solicited  in  terms  that  are  understandable  to 
program  personnel,  must  be  converted  into  a  quantitative 
covariance  value  for  risk  assessment, 

Direction  of  the  Dependence.  One  piece  of  infor¬ 
mation  that  the  analyst  needs  in  order  to  eatimate  the 
covariance  is  the  direction  of  the  component  dependencies. 
The  direction  can  be  implied  simply  by  asking  the  program 
specialist  how  the  extreme  cost  outcomes  for  one  component 
(high  or  low)  relate  to  the  extreme  cost  outcomes  for  the 
other  component.  The  high  and  low  costs  have  the  same 
meaning  here  as  they  did  in  the  variance  estimating  techni¬ 
ques  in  Chapter  2,  If,  in  the  specialist's  judgement, 
component  cost  outcomes  tend  to  be  complimentary  (high-high 
or  low-low),  then  the  implied  sign  of  the  relationship  is 
positive.  If,  on  the  other  hand,  the  cost  outcomes  are 
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perceived  to  be  generally  opposite  (high-low  or  low-high), 
then  the  relationship  is  assumed  to  be  negative.  The  most- 
likely  cost  outcome  could  also  be  used,  but  the  extreme 
values  give  the  analyst  the  best  insight  into  the  direction 
of  the  component  dependencies. 

Strength  of  the  Dependence .  Program  personnel 
will  not  be  able  to  directly  estimate  the  magnitude  of  the 
covariance  because  the  covariance  lacks  any  intuitive  mean¬ 
ing.  As  mentioned  earlier  in  this  chapter,  covariance  may 
be  expressed  in  units  which  are  nonsensical  and  the  magni¬ 
tude  of  the  covariance  will  fluctuate  with  the  scale  of  the 
units  of  measurement.  Conceivably,  these  problems  could  be 
overcome  if  subjective  inputs  were  solicited  in  terms  of  the 
scaled  covariance,  Rho.  Rho  is  unitless  and  ia  standardized 
on  the  interval  (-1,1),  A  more  appropriate  technique,  how¬ 
ever,  would  be  to  solicit  subjective  Inputs  in  terms  of  the 
coefficient  of  determination,  Rho  squared. 

Winkler  and  Hays  describe  the  coefficient  of  determina¬ 
tion  as  the  proportion  of  total  variation  in  a  variable  that 
is  accounted  for  by  a  linear  relationship  with  another 
variable  (19*633).  For  example,  a  Rho  squared  value  of  ,49 
(Rho-, 7)  indicates  that  the  relationship  between  the  two 
variables  accounts  for  49?  of  the  total  variation  which 
would  occur  if  the  variables  were  independent.  Essentially, 
Rho  squared  and  Rho  are  the  same  measure  of  association 
because  the  magnitude  of  one  can  be  determined  from  the 
magnitude  of  the  other,  Rho  lies  on  the  interval  (-1,1), 
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Rho  squared  Is  always  positive  and  Is  bound  by  (0,1). 

The  advantage  of  using  Rho  squared  to  assess  the 
strength  of  dependency  is  that  it  measures  variation  In 
terms  of  proportions.  Therefore,  Identical  increments  any¬ 
where  along  its  interval  have  the  same  ’explanatory  power’ 
because  each  fixed  Increment  represents  a  constant  propor¬ 
tion  of  total  explained  variation.  For  example,  changes  in 
Rho'squared  from  .1  to  .2  and  from  ,8  to  .9  both  increase 
explained  variation  by  10%.  Rho,  however,  is  not  as  'well- 
behaved'  as  Rhc  squared.  Increases  in  Rho  that  are  near  the 
endpoints  of  its  interval  have  gr.eacer  'explanatory  power' 
than  the  same  increment  near  the  center  of  the  interval. 

For  example,  a  similar  change  in  Rho  from  1.  to  .2  and  from 
.8  to  .9  would  explain  3%  and  17%  more  variation,  respec¬ 
tively.  Program  personnel  are  more  likely  to  give  more 
representative  assessments  of  the  magnitude  of  component 
dependence  if  the  analyst  uses  Rho  squared. 

To  apply  the  coefficient  of  determination,  the  program 
specialist  is  asked  to  rate  the  likelihood  of  the  cost 
relationship  that  he  identified  earlier  for  the  direction  of 
the  dependence.  Any  program  personnel  should  be  able  to 
interpret  a  question  such  as,  "On  the  scale  shown  ir. 

Fig.  3.2,  what  is  the  predictability  of  a  component's  cost 
outcome  (high  or  low),  given  that  the  cost  of  the  other 
component  was  high,  or  low"? 


totally 
tncer tnin 


moderately 

certain 


completely 

certain 
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0  --.1  --.2  —  .3  --.4  --.5  --.6  —  .7  --.8  —  .9  —1.0 


Fig.  3.2  R4  Measurement  Scale 


The  values  on  the  interval  in  Fig.  3.2  measure  the  cost 
association  in  a  manner  that  is  analogous  to  Rho  squared.  ' 
Each  value  on  the  interval  represents  the  proportion  of 
certainty  in  the  cost  dependence  between  components.  This 
is  similar  to  Rho  squared,  which  measures  the  proportion  of 
total  variation  that  is  explained  by  the  relationship  with 
another  variable.  Values  near  0  imply  component  independ¬ 
ence  and  values  near  1  imply  total  linear  dependence.  Once 


the  amount  of  dependence  is  assessed,  the  magnitude  of  i^ho 
is  calculated  by  taking  the  square  root  of  the  number  that 
was  selected  on  Fig.  3.2.  Rho  will  be  used  later  to  .calcu¬ 
late  an  estimate  for  the  covariance. 

An  alternate  method  esti maces  the  strength  of  the  de¬ 
pendency  without  relying  on  any  numerical  input  from  the 
program  specialist.  Using  descriptive  categories,  for  exam¬ 
ple,  the  program  specialist  should  be  able  to  categorize  the 
cost  association  between  components  as  either  weak,  mo¬ 
derate,  strong,  or  none.  The  number  of  categories  could  be 
expanded,  but  for  simplicity,  only  these  four  will  be  con¬ 
sidered  here.  An  assignment  for  the  magnitude  of  Rho  can  be 
made  to  each  descriptive  category.  Devore  ).ends  credibility 
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to  such  an  approach  by  giving  general  descriptive  qualities 
to  various  Rho  values  (5:184),  An  example  o£  these  assign¬ 
ments  are  shown  below  In  Table  3.1. 


. .  Table  3.1  / 

■  •  Rho  Value  Assignments 
Component  Relationship  Rho  Value 


None 

Weak 

Moderate 
'  Strong 


.1  to  .3 
.3  to  ,7 
.7  t 


The  actual  magnitude  used  for  Rho  could  be  the  midpoint  of 
the  appropriate  interval.  The  width  of  each  interval  will 

l  1  ’ 

narrow  as  the  number  of  descriptive  categories  is  increased. 

The  magnitude  of  Rho,  based  on  either  technique, „is 
merged  with  the  assessment  of  the  dependency  direction. 
Combined,  the  two  result  in  a  positive  or  negative  Rho 
value.  An  estimate  of  the  covariance  can  now  be  made  from 
the  Rho  value  and  the  statistical  relationships  which  were 
previously  discussed  with  Eq  (3.19). 


v(Xi,Xj)  «  Rx^  [  Var(Xi)  Var(Xj)  ]' 


(3.19) 


The  covariance  is  calculated  from  the  component  variances 
and  the  Rho  estimate  that  was  detarmined  above. 
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Validation  Methodology 

The  application  of  this  cost-risk  assessment  technique 
to  a  real  system  would  not  provide  an  adequate  basis  for 
validating  the  methodology  because  each  program  represents 
only  a  single  event  from  a  unique  distribution.  The  final 
cost  outcome  for  any  single  program  provides  insufficient 
information  to  test  the  accuracy ,  or  lack  thereof,  of  the 
variance  produced  by  risk  assessment  techniques.  For  exam¬ 
ple,  a  cost  outcome  that  deviates  substantially  from  the 
initial  estimate  could  imply  thet  the  true,  system  cost 
variance  was  underestimated.  This  is  illustrated  in 
Fig.  3.3(a).  However,  it  could  also  be  true  that  the  total 
system  variance  is  accurately  estimated,  as  in  Fig.  3.3(b), 
but  the  'most-likely'  cost  estimate  is  in  error.  Finally, 
both  the  point  estimate  and  the  variance  may  be  accurately 
estimated,  but  the  outcome  is  a  'rare  event',  as  in 
Fig.  3.3(c).  The  problem  with  only  one  data  point  is  that  the 
analyst  will  never  know  the  true  state  of  affairs. 


(a) 


<*) 


Fig.  3.3  a/b/c  True  vs  Hypothesized  Cost  Distributions 
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Likewise,  a  cost  outcome  near  the  mode  of  the  hypothe¬ 
sized  distribution  may  give  the  analyst  a  'warm  feeling' 
which  is  entirely  unwarranted.  The  fact  that  the  outcome 
occurred  near  the  mode  of  the  hypothesized  distribution  says 
very  little  about  the  accuracy  of  the  estimated  total  system 
variance.  Furthermore,  the  outcome  could  be  an  outlying 
observation  from  a  true  distribution  that  differs  signifi¬ 
cantly  from  the  hypothesized  distribution,  Again,  the  evi¬ 
dence  is  inadequate  to  draw  any  reasonable  conclusions. 

Because  the  cost-risk  methodology  can  never  be  vali¬ 
dated,  in  the  true  sense,  by  reel-world  application,  the 
authors  propose  a  simulation  technique  as  a  quasi-validation 
tool.  If  the  subjective  inputs  used  in  the  method  are 
assumed  to  be  correct,  the  effect  of  component  dependence  on 
the  total  system  cost  variance  is  indicated  by  comparing  the 
results  of  the  simulation  with  the  case  where  no  dependen¬ 
cies  exist.  When  errors  in  the  subjective  inputs  are  con¬ 
sidered,  simulation  will  lend  insights  on  the  sensitivity  of 
the  total  system  cost  variance  to  those  errors. 

What  is  Simulation?  Banks  and  Carson  (1:2)  describe 
simulation  as  the  imitation  of  a  real-world  system.  The 
behavior  of  the  real  system  is  studied  by  developing  a  model 
which  describes  the  salient  characteristics  of  the  real 
system  through  mathematical  and  logical  relationships. 

Models  th9t  do  not  use  random  variables  as  inputs  are  deter¬ 
ministic.  These  models  have  a  known  set  of  inputs  which 
result  in  a  unique  output.  A  stochastic  simulation  has  one 
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or  more  random  inputs  and  loads  to  random  model  outputs 
(1:10).  The  simulation  model  proposed  in  this  thesis  will 
generate  repeated  estimates  of  the  total  system  variance 
from  stochastic  inputs  which  measure  component  dependency. 
Multiple  iterations  of  the  simulation  generates  an  estimate 
of  the  true  distribution  of  total  system  variance  for  the 
system  described  by  the  model. 

Description  of  the  Simulation  Model .  The  model  used  in 
this  thesis  simulates  a  generic  system  as  illustrated  in  Fig 
3.4,  where  'n*  is  the  number  of  system  components.  The 
components  are  completed  in  sequence  such  that  each  compo¬ 
nent  has  only  one  precedent  task. 


Component  i 


Component  ; 


Component 


Fig.  3.4  Generic  Sequential  System 


This  simplification  restricts  the  number  of  possible  systems 
to  consider  when  compared  to  a  WBS  configuration.  Given  'n' 
system  components,  a  WBS  hierarchy  has  the  flexibility  of 
describing  the  component  relationships  in  a  variety  of  ways, 
while  the  sequential  system  has  only  one.  Such  a  restric¬ 
tion  is  desireable  because  it  allows  us  to  examine  dependen¬ 
cy  relationships  in  their  simplest  form.  This  simplicity 
ulso  makes  the  system  easier  to  model  because  the  system 
definition  can  be  changed  by  varying  system  size  without 
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having  to  redefine  its  'shape'.  Note,  however,  that  this 
simplification  is  made  for  research  purposes  only.  In  no 
way  does  it  restrict  the  cost-risk  assessment  method  from 
application  to  a  WBS  hierarchy.  As  mentioned  earlier,  the 
simplification  reduces  the  number  of  component  pairs  for  the 
system  to  'n-l'.  This  methodology  can  easily  be  extended  to 
any  WBS  framework  with  potentially  as  many  component  cost 
relationships  as  described  by  Eq  (3.9). 

The  simulation  model  stochastically  generates  Rho 
values  for  the  'n-l'  component  pairings  based  on  a  tri¬ 
angular  distribution  with  (0,1)  bounds.  The  triangular 
distribution  was  chosen  because  it  is  unlmodal  and  easy  to 
manipulate.  With  the  fixed  endpoints,  the  researchers  have 
complete  control  over  the  distribution  by  changing  only  one 
value,  the  mode.  The  mode  of  the  distribution  represents  a 
Rho  value  that  could  likely  result  from  a  program  special¬ 
ist's  assessment  of  component  dependence.  Manipulating  the 
mode  allows  the  researchers  to  vary  the  program  specialist's 
estimates  of  the  component  relationships.  The  mode  applies 
for  all  'n-l'  component  pairs,  but  the  exact  Rho  value 
generated  for  each  pair  will  randomly  deviate  from  the  mode 
according  to  the  distribution.  The  mode  is  specified  in  the 
simulation  as  either  .50,  .707,  or  .866. 

A  uniform  distribution  is  then  used  to  randomly  select 
Rho  values  which  ore  subsequently  given  a  negative  sign  to 
indicate  a  negative  dependency.  The  approximate  proportion 


controlled  by  the  modeler's  selection.  The  proportion  is 
set  to  either  1.0,  .75,  or  .50. 


The  model  then  uses  component  variances,  which  are  heldt 
constant  throughout  the  simulation,  and  the  'n-1'  Rho  values 
to  calculate  the  covariance  terms  for  each  sequential  pair 
of  components.  Total  system  variance  ie  then  calculated  by 
summing  the  variance  and  covariance  terms.  Each  observation 
from  the  simulation  represents  one  total  system  variance 
estimate  using  randomly  generated  Rho  values. 

The  model  is  then  re-executed  sc  that  it  generates  a 
new  series  of  Rho  values  using  the  same  triangular  distribu¬ 
tion  and  proportion  of  negative  Rhos,  Covariance  terms  and 
total  system  variance  ore  recalculated  as  before.  The  'out¬ 
put  of  this  simulation  is  another  estimate  of. total  system 
variance,  Multiple  iterations  are  performed  to  provide  a 
distribution  of  probeble  total  system  variances  under  the 
stated  characteristics  of  the  system. 

Sensitivity  Analysis .  As  mentioned  at  the  beginning  of 
this  chapter,  the  objectives  of  the  validation  effort  are  1) 
to  demonstrate  the  significance  of  component  dependence  in 
cost-risk  assessment,  and  2)  to  verify  the  internal  validi¬ 
ty  of  the  proposed  method.  The  researchers  do  not  believe 
that  the  answers  from  these  two  objectives  will  be  similar 
for  all  systems.  Instead,  modeling  component  dependence  may 
be  more  critical  for  certain  systems.  Similarly,  there  may 
be  some  system  'limits'  beyond  which  the  proposed  method  is 
not  a  desireable  technique.  For  this  reason,  the  simulation 
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model  tests  a  variety  of  system  configurations .  Table  'i.2 
summarizes  those  model  inputs  which  characterize  each  system 
and  the  values  they  will  assume.  Those  values  were  chosen 
in  order  to  investigate  the  broad , spectrum  of  dependence 
scenarios  that  could  occur  in  a  system  acquisition  program. 


■  V*  .  Wi  • 

System  Conf  igurat.-lqn.s 


Model  Parameter 

Svmbol  IT. 

Values 

Assumed 

Mode  of  triangular 
distribution  for  rho 

m 

.5 

.  70  7 

.  866 

Proportion  of  positive  rhos 

P 

1.0 

.75 

.30 

Number  of  components 

n 

5 

50 

100 

Error  in  rho  squared 

e 

.10 

.20 

*  Values  represent  rhos  which  explain  25?, 
75%  of  total  variation. 

■H"1*  To  be  discussed  later  in  this  chapter. 

50%, 

and 

Simulations  will  be  performed  for  all  54  combinations  of 
input  parameters.  The  number  of  iterations  required  to 
provide  an  appropriate  estimate  of  total  system  variance  for 
each  system  configuration  is  documented  in  Appendix  A.  The 
computer  algorithm  and  associated  documentation  to  perform 
these  simulations  is  found  in  Appendix  B, 

Is  Dependence  Important?  Ore.  objective  of  the  valida¬ 
tion  effort  is  to  determine  if  component  dependence  should 
be  an  Important  consideration  in  cost-risk  assess menu.  Spe- 


cifically,  what  impact  does  component  dependency  have  on  the 
total  system  cost  variance?  The  simulation  model  was  de¬ 
signed  to  answer  this  question.  Recall  that  under  the 
independence  assumption,  the  total  system  cost  variance  is 
merely  the  sum  of  all  of  the  component  variances.  The 
simulation  model  sums  the  component  variances  to  attain  the 
total  system  variance  for  the  independent  System.  The  model 
also  simulates  54  different  system  configurations  with  var¬ 
ious  levels  of  component  dependency.  A  distribution  of  the 
total  system  cost  variance  will  be  output  for  each  of  these 
systems.  By  comparing  the  total  system  cost  variance  under 
the  independence  assumption  with  total  system  variance  dis¬ 
tribution  under  the  different  system  configurations,  the 
impact  of  component  dependence  can  be  assessed. 

A  measure  of  significance  is  needed  to  compare  inde¬ 
pendence  with  dependence.  The  ideal  measure  would  be  to 
compare  the  affect  of  both  variances  on  the  confidence 
intervals  for  a  system's  cost  estimate  using  Eq  (3.22). 

Interval  Estimate  ■  Point  Estimate  +  [(k)  (s)]  (3.22) 

In  Eq  (3.22),  l<  represents  the  number  of  standard  errors 
that  must;  be  added  and  subtracted  from  the  point  estimate  to 
arrive  at  the  interval  estimate.  It  1b  a  function  of  the 
probability  distribution  of  cost  outcomes  and  tho  desired 
level  of  confidence  that  the  actual  cost  will  fall  within 
the  interval.  The  standard  error  .is  represented  by  s.  The 
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collective  term,  [ ( k )  ( s) ] ,  is  the  precision  of  the  inter¬ 
val  . 

The  above  confidence  interval  in  Eq  (3,22)  would  bn 
incorrect  if  it  used  the  independent  system  standard  error 
(sj)  when,  indeed,  the  system  has  component  dependence.  The 
dependent  standard  error  (s^)  would  have  to  be  used  to 
provide  the.  true  confidence  interval.  Comparing  the  two 
confidence  intervals  would  demonstrate  the  ultimate  impact 
of  component  dependence  on  system  cost  uncertainty.  This 
measure ,  however,  is  not  available  here  becuuse  no  system 
cost  estimate  was  used,  or  assumed.  This  risk  assessment 
method  simply  assumed  that  a  cost  estimate  is  available  from 
current  estimating  techniques.  Furthermore,  k  requires  an 
assumption  about  the  cost  distribution's  shape,  which  is 
beyond  the  scope  of  this  research. 

Although  this  research  cannot  directly  compare  the  two 
cost  intervals,  it  can  moke  a  direct  comparison  of  the 
interval  precision,  us  shown  in  Eq  (3.23), 

[  (k)  (B|))  j  /  l  (k)  ( Sj)  j  (3.23) 

Because  k  hao  a  constant  value,  Eq  (3.23)  simplifies  to 
[(s^)  /  (a,)].  The  magnitude  of  this  ratio  indicates  the 
relative  impact  of  dependence  r e J a t i on  a h i ps  on  the  system's 
confidence  intorvul.  The  ratio  indicates  t.o  what  extent  the 
precision  of  the  interval  is  over  or  understated.  If  the 
ratio  is  close  to  one,  component,  dependency  is  not  signifi- 


cant  and  the  difference  between  the  cost  Interval  using  s ^ 
and  Sj  is  small.  Ae  the  ratio  moves  away  from  one.  in 
either  direction,  the  component  dependency  relationships  are 
more  significant.  Under  these  conditions,  confidence  inter¬ 
vals  for  weapon  system  cost  estimates  become  increasingly 
unreliable  using  the  independent  standard  error. 

Because  this  research  focuses  on  vafien.c.es  rather  than 
standard  errors,  the  authors  will  use  the  ratio  of  the  two 
total  ayspem  coat  variances  aa  the  measure  of  significance. 
The  relationship  between  the  ratio  of  the  variances  and  the 
ratio  of  the  standard  errors  la  shown  in  Eq  (3.24). 
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(3.24) 


The  mean  of  the  total  eyateth  cost  variance  distribution  will 
be  used  as  the  e  q  for  each  system.  The  accuracy  of  this 
value  in  estimating  the  true  mean  of  the  distribution  is 
discussed  in  Appendix  A.  The  authors  consider  1.2  an 
overestimate  and  .SO  an  underestimate  of  the  precision 
[(k)  (s) ]  In  Eq  (3.23)  to  be  acceptable.  Therefore,  any 
ratio  of  total  system  cost  variance  that  is  between  .64  and 
1.44  meets  the  acceptance  criteria.  For  those  systems,  the 
affects  of  component  dependence  is  not  considered  to  be 


jective  of  the  validation  effort  is  to  determine  the  inter¬ 
nal  validity  of  the  cost-risk  assessment  method.  The  inter¬ 
nal  validity  indicates  the  ability  of  the  analyst  to  predict 
the  total  system  cost  variance  for  a  system  with  component 
dependence  when  the  program  specialist's  subjective  assess¬ 
ments  are  erroneous.  Realistically,  the  specialist  will  not 
be  able  to  estimate  the  strength  of  dependencies  accurately. 
Additional  error  will  be  induced  by  the  method  which  con¬ 
verts  the  specialist's  Inputs  into  a  Rho  value.  This  is 
especially  true  with  the  flexibility  of  Rho  assignments  to 
descriptive  categories,  As  a  result,  errors  in  the  Rho 
estimate  will  affect  the  calculation  of  total  system  cost 
variance.  The  simulations  described  so  far  hove  assumed 


that  the  analyst  has  an  accurate  Rho  estimate.  We  assumed 
that  the  mode  of  the  triangular  distribution  and'  the  propor¬ 
tion  of  negative  dependencies  were  correctly  identified, 

The  simulation  wdl,i  output  a  distribution  of  the  total 
system  variance  under  this  assumptiong j  wo  refer  to  it  as  the 
'normal'  case.  However,  this  distribution  would  be  affected 
by  errors  in  estimating  Rho,  as  demonstrated  in  Fig.  3.5. 
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Fig.  3.5  Cost-risk  Assessment  with  Rho  Error 


The  simulation  model  evaluates  the  impact  of  the  error  in 
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subjective  inputs  by  placing  upper  and  lower  bounds  on  the 
Rhos  that  are  randomly  generated.  These  bounds  represent 
the  analysts  confidence  in  the  Rho  estimate.  If  an  analyst 
feels  uncertain  about  his  estimate  of  Rho,  he  could  place 
bounds  around  his  estimate  and  also  calculate  the  total 
syctem  variance  Using  these  bounds  as  a  way  of  evaluating 
the  potential  effect  of  his  estimating  error. 

A , method  to  bouhrl  Rho  would  be  to  bound  the  percent  of 
total  variation  represented  by  Rho  squared.  For  example, 
Suppose  the  analyst  feels  that  Rho  is  approximately  .7, 
which  equates  to  a  Rho  squared  of  ,49.  This  figure  indi¬ 
cates  that  the  relationship  between  the  two  variables  ac- 
ebuntfl  for  49%  of  the  total  variation  which  would  'occur  if 
the  variables  were  completely  independent.  If  the  analyot 
felt  that  the  correct  total  variation  should  not  vqry  by 
more  than  plus  or  minus  five  percent,  the  upper  and  lower 
bounds  on  Rho  squared  would  correspond  to  54%  and  45%, 
respectively.  Taking  the  square  roots  of  these  percentages 
results  in  upper^and  lower  bounds  on  Rho. 

Rho  squared  error  is  introduced  into  the  simulation 
using  the  parameter  'o’.  Two  values  will  be  assigned,  .1 
and  ,2.  The  .1  value  was  the  basis  of  the  above  example, 
where  the  bound  on  Rho  squared  for  a  possible  10%  error  (+_ 
5%)  In  total  explained  variation.  The  simulation  model 
'splits'  the  ’ p '  value  equally  around  the  Rho  squared  for 
the  'normal'  case.  Any  bounds  which  foil  outside  of  the  0 
and  1  Interval  for  Rho  squared  ore  truncated  to  0  or  1. 
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The  1  over  and  upper  bounds  on  Rho  can  be  used  to  calcu¬ 
late  a  total  system  variance  which  la  consistently  higher  or 
lower  than  the  total  system  variance  for  the  'normal'  case. 
Recall  that  the  total  system  variance  is  calculated  by 
summing  all  system  variance  and  covariance  terms  There¬ 
fore,  given  constant  variances,  the  total  system  variance  is 
affected  by  the  changing  only  values  of  the  covariance 
terms.  The  covariance  was  defined  in  Eq  (3.19)  as 
Cov(XitXj)  ■  Rx  x  [  Var(X^)  Var(Xj)  .  From  this  equa- 

tion,  the  covariance  for  each  component  pair  is  maximized 
when  positive  Rhos  assume  their  highest  magnitude  and  nega¬ 
tive  Rhos  assume  their  lowest.  Similarly,  covariance  is 
minimized  when  positive  and  negative  Rhos  assume  their  low¬ 
est  and  highest  values,  respectively.  If  the  upp*.  and 
lower  Rho  values  for  each  component  pair  are  used  accord¬ 
ingly  in  the  computation  of  the  total  system  variance,  the 
resultant  output  distributions  reveal  the  highest  and  lowest 
possible  deviations  of  total  system  variance  from  the  'nor¬ 
mal'  case. 

Large  overlaps  between  the  'normal'  and  'low'  and  be¬ 
tween  the  'normal'  and  ’high'  distributions  indicate  only 
minor  sensitivity  to  Rho  estimating  error.  If  the  model 
produces  'low'  and  'high'  distributions  that  are  close 
enough  to  the  'normal  case',  then  the  methodology  may  be 
applicable  to  real  systems  with  predominantly  similar  char¬ 
acteristics.  However,  the  model  may  reveal  Lhat  predictions 
of  total  system  cost  variance  deviate  unacceptably  when  Rhos 
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are  slighlty  insecure fe .  This  tells  us  that  the  methodology 
requires  further  refinement,  or  that  it  should  be  selec¬ 
tively  applied  to  only  the  most  'forgiving'  systems. 

Overlap  of  the  'low'  and  'normal1  distributions  will  be 
measured  by  counting  the  number  of  observations  on  the  'low' 
distribution  that  appear  above  the  5,  10,  and  15  percentiles 
of  the  'normal'  case.  Overlap  of  the  'high'  and  'normal' . 
distributions  will  be  measured  by  counting  the  number  of 
observations  on  the  'high*  distribution  that  fall  below  the 
95,  90,  and  85  percentiles  of  the  'normal'  case.  This 
technique  provides  the  researchers  with  a  standard  measure 
of  overlap  for  the  54  systems  without  making  any  assumptions 
about  the  shape  of  their  three  variance  distributions.  It 
also  provides  a  'well-behaved'  measurement  that  is  not  sen¬ 
sitive  to  the  location  of  'rare  events'  in  the  'normal' 
case. 

A  decision  rule  is  needed  to  determine  what  constitutes 
'significant'  overlap  within  the  six  percentiles  on  the 
'normal'  distribution.  The  authors  feel  that  the  standards 
in  Table  b.3  are  reasonable.  Our  intention  is  to  determine 
how  well  each  system,  on  the  whole,  overlaps  the  'normal' 
case.  The  above  criteria  yields  six  standards  for  each  of 
the  54  systems,  however,  the  authors  do  not  want  a  prolifer¬ 
ation  of  standards  to  'mask'  the  results.  For  this  reason, 
we  choose  three  criteria  to  measure  each  system's  perform¬ 
ance  against  the  standards. 


Table  3.3 

Acceptance  Standards 

X  of  ' low ’ 

%  of  'high' 

1  normal ' 

observations 

observations 

Dercentile 

above  %tile 

below  Stile 

.05 

70T~~' 

- 

.10 

60 X 

— 

.15 

50% 

- 

.95 

- 

70% 

.90 

— 

60% 

.85 

- 

50% 

Criteria  #1  is  Lha  mast  stringent  of  the  three.  It 
describes  only  those  systems  which  meet  all  six  of  the 

p  •  •  •  • 

standards  in  Table  3.3.  This  criteria  Identifies  those 
systems  for  which  the  proposed  risk  assessment  methodology 
has  strong  predictive  ability.  Criteria  # 2  describes  those 
systems  with  both  ’high'  and  ’low'  distributions  that  meet 
at  least  two  of  their  three  standards.  In  other  words, 
neither  the  'high'  nor  'low'  distributions  can  fail  to  meet 
more  than  one  standard.  The  risk  assessment  methodology  i3 
moderately  able  to  predict  total  system  cost  variance  for 
these  systems.  Those  systems  which  met  Criteria  #1  are 
excluded  from  this  category.  Criteria  §3  describes  all 
other  systems.  For  these  systems,  the  proposed  methodology 
exhibits  Its  weakest  application  potential. 

Because  the  selection  of  standards  is  subjective,  the 
researchers  feel  that  a  second  sot  of  standards  will 
strengthen  the  reliability  of  the  findings.  The  second  set 


of  standards  will  be  similar  to  those  in  Table  3.3,  except 
that  the  percentage  of  overlap  for  each  percentile  will  be 
'loosened'  by  20  percent  (50%,  40%,  30%).  Criterion  #1,  § 2 
and  #3  will  also  he  applied  to  this  set  of  standards.  Hope¬ 
fully,  this  approach  will  lend  better  visibility  to  any 
trends  that  might  not  emerge  with  a  more  rigid  standard. 

Conclusion 

1  1  This  chapter  has  outlined  the  methodology  that  will  be 

used  to  Incorporate  component  dependency  into  an  aaseaament 
of  total  system  cost  variance.  The  underlying  statistical 
framework  was  developed  as  well  as  a  technique  to  apply 
those  concepts.  A  validation  methodology  of  the  risk  as¬ 
sessment  technique  was  proposed.  The  following  chapter 
analyzes  the  results  of  this  methodolgy. 
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I V .  Analysis 


This  chapter  will  apply  the  research  methodology 

developed  in  the  previous  chapter  to  assess  the  results  of 

* 

»  the  simulation,  The  data  that  is  generated  will  be  used  to 

v'  assess  the  heed  for  the  incorporation  of  component  depend- 

* 

ence  in  cost-risk  assessment.  The  final  portion  of  the 
chapter  is  devoted  to  a  discussion  of  the  internal  validity 
of  the  model  and  its  implications  for  the  cost  analyst. 

The  simulation  was  performed  for  each  of  the  possible 
system  configurations  in  Chapter  3,  Table  2.  The  output 
files  of  the  Fortran  simulation  (OUT!  -  0UT54)  were  input 
into  the  BMDP  statistical  package  which  sorted  the  total 
system  cost  variances  for  the  ’low',  'normal',  and  'high' 
distributions,  plotted  histograms  of  these  distributions, 
and  provided  pertinent  statistical  information  (6: 74—78, 112— 
113,  124-132).  The  BMDP  filenames  (RUN1  -  RUN54)  are  listed 
in  Table  4.1,  below,  with  a  list  of  the  parameters  that  were 
modeled.  The  BMDP  code  is  found  in  Appendix  C.  The  histo¬ 
grams  are  provided  in  Appendix  D. 

• 

R u t i o  of  Variances 

*  As  mentioned  in  Chapter  3,  the  ratio  of  variances  was 

* 

used  ns  a  measure  of  the  significance  of  component  dependence 
in  co st -risk  assessment.  Table  4.2  gives  the  total  cost 
variance  assuming  independence  for  each  system  and  summarizes 
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the  calculations  for  the  ratio  of  variances 


Table  4. 

System  Configurations 

1 

by  Run 

Number 

e 

■  .  2 

n  ■ 

5 

e 

■  .  1 

RUN# 

m 

£ 

RUN# 

£ 

£ 

1 

.500 

1.0 

10 

.500 

1.0 

2 

.500 

.75 

11 

.500 

.75 

3 

.500 

.50 

12 

.500 

.50 

4 

.707 

1.0 

13 

.707 

1.0 

5 

.707 

.75 

14 

.707 

.75 

6 

.707 

.50 

15 

.707 

.50 

7 

.866 

1.0 

16 

.866 

1.0 

8 

.866 

.75 

17 

.866 

.75 

9 

.866 

.50 

18 

.666 

.50 

e 

-  .2 

n  ■ 

50 

e 

*  .  1 

RUN# 

rn 

£ 

RUNf 

m 

£ 

19 

.500 

1.0 

28 

.500 

1.0 

20 

.500 

.75 

29 

.500 

.75 

21 

.500 

.50 

30 

.500 

.50 

22 

.707 

1.0 

31 

.707 

1.0 

23 

.707 

.75 

32 

.707 

.75 

24 

.707 

.50 

33 

.707 

.50 

25 

,  866 

1.0 

34 

.866 

1.0 

26 

.866 

.75 

35 

.866 

.75 

27 

.866 

.50 

36 

.866 

.50 

62 


Table  4.1  (continued) 

System  Configurations  by  Run  Number 


e 

=  .  2 

n  * 

1 00 

e  **  . 

1 

RUN# 

m 

£ 

RUN# 

in 

£ 

37 

.500 

1.0 

46 

.500 

1.0 

38 

.500 

.75 

47 

.500 

.75 

39 

.500 

.50 

48 

.500 

.50 

40 

.707 

1.0 

49 

.707 

1.0 

41 

.707 

.75 

50 

.707 

.75 

42 

.707 

.50 

51 

.707 

.50 

43 

.866 

1.0 

52 

.866 

1.0 

44 

.866 

.75 

53 

.  B66 

.75 

45 

.  866 

.50 

54 

.866 

.50 

Based  on  the  acceptance  criteria  described  in 
Chapter  3,  component  dependence  is  considered  significant 
when  the  ratio  of  variances  is  less  than  .64  or  greater  than 
1.44.  Table  4.2  reveals  that  component  dependence  is  sig¬ 
nificant  in  a  majority  of  the  systems  that  were  modeled. 
Closer  examination  of  the  table  shows  several  trends.  The 
first  trend  has  to  do  with  the  number  of  components  in  the 
system,  'n'.  The  rows  of  Table  4.2  show  identical  system 
configurations  except  for  the  number  of  components.  The 
ratios  across  each  row  tend  to  be  clustered  near  the  same 
value.  These  results  indicate  that  system  si/e  does  not 
significantly  affect  the  movement  of  total  system  cost  var- 
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iance  away  from  the  independent  case.  This  implies  that  the 
number  of  system  components  should  not  affect  the  decision 
to  model  component  dependence  in  cost- risk  assessments. 

Table  4. 2 

Ratio  of  Variances w 


n  * 

VAR  - 

5 

336 

.  ...  n  « 

V,AR  - 

50  . 

2339 

"h  ■ 

VAR  - 

100 

4616 

RUN 

Ratio 

RUN  : 

Ratio 

RUN 

Ratio 

.1410 

1.71 

19428 

1.81 

37446 

1.85 

2411 

1.38 

'  20429 

1.40 

38447 

1.43 

3  &  1 2 

1.03 

21430 

•  1.00 

39448 

1.00 

4413 

1.82 

22431 

1.92 

40449 

1 . 97 

5414 

1.50 

2343,2 

1.46 

41450 

1.49 

6415 

1.03 

24433 

1.00 

42451 

1.00 

7416 

1.90 

25434 

2.01 

43452 

2,06 

8417 

1.54 

26435 

1.50 

44453 

1.53 

9418 

1.03 

27436 

1.00 

45454 

1,00 

^  Because  the  calculations  for  this  table  were  based  on  the 
1  norinu  1  1  cose ,  Rho  estimation  error  does  not  affect  the  ratios 
for  these  runs. 


Caution  should  be  taken,  however,  in  generalizing  this 
conclusion  to  systems  with  a  WBS  hierarchy.  The  conclusion 
that  system  size  Is  insignificant  seems  to  be  more  limited 
to  the  simplified  systems  that  this  research  addressed.  To 
illustrate,  consider  a  30  component  system  whose  depend- 


encies  have  been  simplified  by  the  linear  dependence  as¬ 
sumption.  The  ratio  of  the  dependent  and  independent  vari¬ 
ances  for  this  system  is  shown  in  Eq  (4.1). 

30 

Ratio  •  [  5>ar(X,  ) 

i-1  1 30  30 

+  2  £Cov(X  j  /  XVar(Xi)  (4.1) 

i «2  i«l 

Suppose  that,  the  system  size  is  increased  by  one  component. 
The  new  ratio  of  the  cost  variances  is  shown  in  Eq  (4.2). 

30  30 

Ratio  -  [£Var(X.)  +  2  £Cov(X,  ,,X.)  +  Var(X«.) 
i«l  1  i«2  J 

30 

+  2  Cov(X30lX31)  ]  /  [  3CV«r(.Xi) 

+  Var(X31)  (4.2) 

The  numerator  of  the  ratio  is  increassd  by  the  variance  of 
the  new  component  plus  twice  the  covariance  between  the  new 
component  and  its  Immediate  predecessor.  The  denominator  of 
the  ratio  is  increased  by  the  variance  for  the  new  compo¬ 
nent  . 

System  size  has  a  greater  affect  on  the  ratio  of  the 
two  cost  variances  whon  the  system  is  defined  by  a  WBS 
hierarchy.  Eq  (4.3)  shows  the  ratio  for  the  30  component 
system  which  has  been  enlarged  by  the  additional  component. 
Comparing  Eqs  (4.3)  and  (4.2)  shows  that  adding  another 
component  may  have  a  significantly  greater  affect  when  there 
are  no  constraints)  placed  on  the  covariance  terms. 


30  30  30 

Ratio  -  [  £Var(X.)  +  £  £Cov(X,,X.)  +  VarCX.,) 
i-1  1  i-lj-1  1  J  31 

30  30 

+  £Cov(X  X  x)  +  £Cov(X3]  ,X  )  ] 
i  ■  1  J  =  1  J 

30 

/  [  £  Var(X^)  +  Var(X31)  ]  (4,3) 

i«  1 

The  denominator  of  both  equations  increased  by  the  same 
amount,  however,  the  numerator  of  Eq  (4.3)  may  increase 
significantly  more  than  the  numerator  of  Eq  (4,2).  The 
numerator  of  Eq  (4.3)  includes  not  only  another  variance 
term,  but  also  the  sum  of  60  covariance  terms,  as  opposed  to 
2  additional  covariance  terms  in  Eq  (4.2).  This  could  make 
a  significant  difference. 

Since  the  results  in  Table  4.2  are  not  sensitive  to 
system  size,  only  Column  1  of  the  table  will  be  used  in 
order  to  simplify  the  remaining  discussion.  Although  only 
the  n»5  system  will  be  illustrated,  the  following  trends 
also  apply  to  the  systems  with  50  and  100  components. 

The  first  trend  deals  with  the  relationship  between  the 
proportion  of  positive  rhos,  ’ p  * ,  and  the  mode  of  the  tri¬ 
angular  distribution,  'in',  which  produced  the  Rho  values. 
Table  4.2,  Column  1  is  reproduced  below  along  with  the 
corresponding  values  of  'm'  and  ’ p  * .  High  proportions  of 
posilive  Rhos  consistently  produced  higher  deviations  of 
total  system  cost  variance  from  the  independent  case.  This 
is  evident  by  the  high  ratios  for  RUN1,  RUN4,  and  RUN7  when 
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Table 

m  and  p  Values 

4.3 

for 

RUN1-RUN9 

Run 

Ratio 

m 

P. 

1 

1.71 

.500 

o 

o 

• 

1  4 

2 

1.38 

.500 

.75 

3 

1.03 

.500 

-  .50 

4 

1.82 

.  707 

1, 00 

5 

1.50 

1  '  • 

.707 

.75 

6 

1.03 

.707 

.50 

7 

1.90 

.866 

o 

o 

* — 1 

8 

1.54 

.866 

.75 

9 

1.03 

.866 

.50 

As  this  proportion  decreases,  so  does  the  ratio  of  vari¬ 
ances.  A  decrease  in  ' p  *  from  1.0  to  .75  changes  the  ratio 
for  RUN4  from  1.8  to  1,5.  The  ratios  for  RUNs  1  and  7  also 
changed  by  approximately  ,3.  A  reduction  in  the  proportion 
from  .75  to  .50  changes  the  ratio  for  RUN  5  from  1.5  to  1.0. 
A  similar  decrease  is  evident  in  RUNs  2  and  8. 

These  results  can  be  explained  by  recalling  that  the 
only  difference  between  the  dependent  and  independent  total 
system  cost  variances  is  the  inclusion  of  the  covariance 
terms.  If  the  sum  of  the  covariance  terms  significantly 


deviates  from  zero,  a  difference  in  the  ratio  of  the  two 


variances  will  result.  The  sum  of  the  covariance  terms  is 


very  sensitive  to  the  proportion  of  those  covariances  which 
are  positive.  When  the  proportion  is  large,  there  are 
relatively  few  negative  covariance  terms  to  offset  the  in¬ 
crease  in  the  total  system  cost  variance  caused  by  the 
positive  covariances.  As  shown  in  Table  4.3,  when  half  of 
the  Rhos  were  positive,  the  results  closely  approximate  the 
independent  system  for  all  values  of  the  mode  used  to  gen¬ 
erate  cheRhos, 

Comparison  of  RUNs  (1-3),  RUNs  (4-6),  and  RUNs  (7-9) 
demonstrates  the  influence  of  the  mode  of  the  triangular 
distribution  on,  the  ratio  of  variances.  Higher  value's  of 
Rho  consistently  produce  greater  deviations  from  the  inde¬ 
pendent  total  system  cost  variance.  These  results  are 
explained  by  Recalling  she  relationships  in  Eq  (3,19).  For 
component  pairs  and  their  associated  varianceo 


Cov(Xi,Xj) 


[  Var(Xi)  Var ( X j  )  ]* 


(3.19) 


The  magnitude  of  Rho,  as  modeled  by  the  mode,  determines  the 
magnitude  of  the  covariance  terms.  With  other  model  parame¬ 
ters  held  constant,  large  magnitudes  of  Rho  (+  or  -)  drive 
the  magnitude  of  the  sum  of  the  covariances.  As  mentioned 
above,  aa  the  sum  of  the  covariances  deviates  from  zero,  the 
ratio  of  total,  system  cost  variances  will  deviate  from  1. 

Stepwise  changes  in  'in'  from  .5  to  .707  and  from  .707 
to  .866  increase  the  ratio  of  the  variances  between  runs  by 


approximately  .1.  For  example,  the  ratio  for  RUNs  1,  4,  and 
7  are  1.7,  1.8,  and  1.9,  respectively.  Each  of  these  step¬ 
wise  changes  in  the  mode  represent  ar  increase  in  total 
explained  variation  of  25%  (Reference  Chapter  3,  Table  1 ) . 

For  the  relevant  ranges  of  the  proportion  of  positive 
Rhos  and  modal  values  chosen  in  this  simul  ation ,  total 
system  cost  variance  is  more  sensitive  to  the  proportion  of 
positive  Rhos,  '  p ,  than  to  the  range  of  mode  values.  ,  Table 
4.3  shows  that  varying  'p'  from  its  lowest,  to  hi ghest .  value 
(in  increments  of  25%)  increases  the  variance  ratio  by 
approximately  .8.  Varying  'm'  from  its  lowest  to  highest 
values  (in  increments  of  25%  of  explained  variation)! 
increases  the  ratio  by  only  ,2.  This  result  has  a  favorable 
implication  for  the  cost-risk  assessment  method.  In  the 
'real  world',  determination  of  the  direction  of  component 
dependency  should  be  easier  to  subjectively  assess  than  the 

i 

exact  magnitude  of  the  dependence  relationship, 

Internal  Validity 

The  high  and  .low  overlap  percentages  generated  by  the 
validation  model  are  displayed  in  Appendix  E.  These  values 
were  compared  to  the  standards  and  acceptance  criteria  that 
were  presented  in  the  previous  chapter.  The  results  of  the 
first  standard  (70-60-50)  are  displayed  in  Table  4.4, 

The  purpose  of  this  section  :Ls  to  determine  the  ability 
to  predict  total  system  cost  variance  when  subjective  esti¬ 
mates  for  Rho  squared  are  in  error.  This  parameter,  ' e '  , 
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was  given  two  values,  .2  and  .1,  As  expected,  more  runs  had 
acceptable  overlap  when  e».l  than  when  e».2  .  This 
makes  sense  intuitively,  since  one  would  expect  a  better 
estimate  of  the  total  system  cost  variance  when  the  value 
for  Rho  was  estimated  with  greater  accuracy.,  In  Table  4.4, 
Matrices  2,  4,  and  6  were  generated  when  the  Rho  squared 
error  was  .1  and  Matrices  1,  3,  and  5  were  generated  with 
the  error  at  ,2.  There  are  18  of  27  runs  that  met  Criteria 
#1  when  ,  e»,,l  .  Only.  8  of  '  27  runs  have  met  this  criteria 

when  p-.2  ,  As  the  program  personnel's  ability  to  subjec¬ 

tively  estimate  component  dependence'  increases,,  the  ability 
of  the  cost-risk  assessment  method  to  accurately  estimate 

total  system  cost  variance  increases  significantly. 

2 

While  R  error  is  the  parameter  used  to  assess  internal 
validity,  the  other  model  parameters  influence  the  applica¬ 
bility  of  the  method.  Looking  again  at  Table  4.4,  the  first 
column  in  each  matrix,  except  Matrix  2,  shows  a  consistent 
weakness  in  predicting  total  system  cost  variance.  Whether 
R^  error  is  .1  or  .2,  the  cost-risk  assessment  method  loses 
predictive  capability  as  the  proportion  of  positive  depend¬ 
encies  approaches  1.0,  The  number  of  components  in  the 
system  also  appears  to  affect  the  accuracy  of  the  method¬ 
ology.  When  1 n  *  is  low,  the  predictive  capability  of  the 
1 

method  is  at  its  best.  This  is  shown  by  the  large  number  of 
systems  in  Matrices  1  and  2  that  meet  Criteria  #  1  . 


Table  4.4 


Internal  Validity  Result  Matrix 
Standard:  70-60-50 


MATRIX  1  MATRIX  2 


n-5 

P 

n»5 

P 

e»  .  2 

1.0 

.  75 

.50 

e- .  1 

1.0 

.75 

.50 

.  500 

,1 

2. 

' 

3* 

.  500 

10* 

11* 

12* 

m 

.707 

4 

5* 

6* 

m  .  707 

13* 

14* 

15* 

.866 

7 

— j 

8* 

9* 

.  866 

16* 

, 

17* 

18* 

MATRIX 

3 

MATRIX  4 

n-50 

P 

n-50 

P 

e-  .  2 

1.0 

.75 

.50 

e« .  1 

1.0 

.75 

.50 

.  500 

19 

20 

21 

.  500 

28 

29* 

gg 

m 

.  707 

22 

23 

24* 

m  .707 

31 

32* 

.  866 

25 

J 

26** 

27* 

.  866 

34 

36* 

MATRIX 

5 

MA'IRIX  b 

ii-  1  00 

P 

n  =  l  00 

P 

e  =  .  2 

1 .0 

.  7  5 

.  30 

e= .  1 

1  .0 

.  75 

.  50 

.  500 

3  7 

38 

39 

.  500 

4  6 

.  - 

48 

HI 

.  707 

40 

4  1 

42 

m  .707 

4  9 

50** 

51* 

.866 

'Vl  1 

— 

44 

4  5 

,  86b 

5  2 

r)  » 

- 

54* 

_ 

-  mee  t.  s  Or  i  t.i.  r  i  a  H  1 
*  *  in  eels  Criteria 

not.-:  no  entry  means  the  system  meets  criteria  if'i 


However,  as  system  size  increases,  the  number  of  runs  that 
meet  the  criteria  decreases  in  Matrices  3  and  4,  and  again 
In  Matrices  5  and  6.  Finally,  the  strength  of  the  component 
dependencies,  as  described  by  the  mode,  appears  to  have  a 
limited  effect  on  predicting  total  system  cost  variance.  As 
'm'  increases  from  .5  to  .866,  in  all  six  matrices,  slightly 
more  runs  meet  Criteria  #1. 


Table  4.4  reveals  that  only  a  limited  number  of  systems 


met  Criteria  #2.  So  few,  in  fact,  that  little  information 
regarding  trends  in  the  performance  of  the  method  can  be 
ascertained. 

Criteria  #3  shows  the  systems  for  which  the  method  has 
limited  ability  to  accurately  predict  total  system  cost 
variance.  Since  so  few  systems  met  the  second  criteria, 
systems  described  by  Criteria  #3  are  basically  the  reverse 
of  those  described  by  Criteria  #1,  The  same  trends  pre¬ 
sented  in  the  previous  discussion  hold  true  and  will  not  be 
rci to  rated. 

The  results  of  the  comparison  of  the  54  systems  to  the 
second  standard  are  shown  in  Table  4.5.  Notice  that  the 
trends  mentioned  iri  the  previous  two  paragraphs  hold  true, 
i'his  consistency  adds  to  the  reliability  of  the  conclusions. 
One  noteworthy  occurrence  is  the  drastic  increase  in  the 
number  of  systems  that  meet  Criteria  #1  for  Matrices  3  and 
() .  A  lowering  o  1  t  he  .standard  tripled  the  number  of  systems 
that  in  <’  l  the  first  criteria. 
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MATRIX  1 
n»5 
e* .  2 

.500 
m  .707 
.866 

MATRIX  3 
n*50 
c  = .  2 

.500 
m  .707 

.866 

MATRIX  5 
n  -  1  00 

i.  =• .  2 

.  500 
in  .707 

.  866 


Table  4.5 

Internal  Validity  Result  Matrix 


Standard!  50-40-30 


MATRIX  2 
p  n«5 

1.0  .75  .50  e-.l  1 

— - — r  -r -—i  r~— 

1#  2*  3*  .500  I< 

4**  5*  6*  m  .  707  II 

7*  8#  9*  .866  ll 

MATRIX  4 
p  n»50 

1.0  .75  .50  e-.l  1 

19  20*  21*  .500  2i 

22  23*  24*  m  .707  3 

23  26*  27*  .866  3' 

MATRIX  6 

p  n- 1 00 


44*  45* 


11*  12* 


15* 


28 

29* 

30* 

31* 

32* 

33* 

34* 

35* 

36* 

.500 

46 

47* 

48* 

.  707 

49 

50* 

51* 

.866 

52 

53* 

54* 

*  meets  Criteria  rY  1 
*  *  meet  s  Criteria  H2 

nuLe:  no  entry  me  a  ns  the  system  meets  criteria  H'i 


[»  spite  of  the  Increase  in  the  number  of  systems  meeting 
Criteria  # 1  in  Matrices  3  and  6,  no  systems  meet  the  cri¬ 
teria  for  Column  #1  in  these  matrices  when  the  proportion  of 
positive  Rhos  was  1,0.  This  reinforces  the  earlier  point 
that  the  cost-risk  assessment  method  has  poor  predictive 
capabilities  when  p».l 

Consistent  with  Table  4.4,  Table  4.5  has  only  two 
systems  meeting  Criteria  #2.  The  minimal  number  of  systems 
meeting  this  criteria  is  significant.  Since  so  few  systems 
meet  this  criteria  of  moderate  predictive  ability,  one  con¬ 
clusion  is  that  the  method  tends  to  have  either  very  strong 
or  very  limited  predictive  capabilities.  This  is  true  even 
when  the  two  standards  used  to  define  acceptable  overlap 
were  significantly  different.  This  conclusion  is  over¬ 
whelmingly  supported  by  the  data. 

In  general,  the  cost-risk  assessment  method,  incorpo¬ 
rating  component  cost  dependence,  predicts  total  system  cost 
variance  with  relatively  good  accuracy  when  the  error  in  Rho 

squared  is  .1.  It  does  not  do  us  well  when  the  error  is  .2. 

2 

Values  of  'o'  greater  than  .2  would  indicate  that  R  values 

are  in i  s-es t ima t i ng  more  than  20 %  of  the  total  system  varia- 

2 

lion.  To  estimate  total  system  cost  variance  when  R  error 
in  grenter  than  .2  would  bo  accepting  entirely  too  large  an 
error  into  the  methodology.  When  the  percentage  of  positive 
covariance  values  approaches  1.0,  the  me t hod  consistently 
predicts  poorly.  However,  systems  with  all  positive  cost 
dependencies  are  highly  unlikely.  In  the  researcher's 


opinion,  a  much  more  likely  scenario  would  have  50%  to  75% 
positive  cost  dependencies.  With  this  situation,  the  pre¬ 
dictive  capabilities  of  the  cost-risk  assessment  method  are 
much  better.  Another  expected  result  is  the  effect  of 
system  size  on  the  predictive  capabilities  of  the  method. 

As  'n'  increases,  the  estimation  error  for  R  is  compounded 
and  estimates  of  total  system  cost  variance  are  further 
distorted.  The  mode  value  influences  total  system  cost 
variance  in  small  amounts.  The  higher  the  value,  the  higher 
the  error  in  the  estimate  of  total  system  cost  variance. 
Finally,  the  lack  of  system  configurations  that  fell  into 
the  second  criteria  for  both  standards  indicates  that  there 
are  clearly  situations  in  which  the  method  has  strong  pre¬ 
dictive  abilities  and  situations  in  which  the  method  has 
weak  predictive  capabilities.  This  result  enhances  the 
value  of  the  method  because  it  defines  the  situations  where 
the  method  can  be  used  with  confidence.  The  pronounced 
division  in  the  number  of  systems  meeting  Criterion  #1  and 
#2  may  be  caused  by  the  rather  ’crude'  incrementing  of  the 
model  parameters  chosen  for  the  sensitivity  analysis. 
Modelling  smaller  incremental,  changes  in  these  parameters 
may  identify  more  situations  with  moderate  predictive  cap- 
nbi  li. Ly.  It  is  interesting  to  note  that  the  method  may  even 
be  usable  In  situations  where  it  has  poor  predictive  capa¬ 
bilities.  As  the  proportion  of'  positive  covariance  terms 
a  pp  roue  hi-fi  1,0,  the  method  predicts  total  system  cost  vari¬ 
ance  with  less  accuracy.  The  results  from  the  ratio  of  the 


variances  indicate  Lhnt  the  incorporation  of  component  de- 
pedency  becomes  most  important  when  the  proportion  of  pos¬ 
itive  component  dependencies  approaches  1,0.  The  method  is 
needed  the  most  when  it  is  least  effective.  However,  even 
though  the  methods  predictive  abilities  are  not  at  their 
best,  the  method  is  still  accounting  for  an  increase  in  the 
total  system  cost  variance  that  is  not  accounted  for  when 
independence  is  assumed. 
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V .  Conclusions  and  Recommendations 


Conclusions 

The  results  from  this  research  indicate  that  component 
cost  dependency  should  be  an  important  consideration  in  the 
cost-risk  assessment  process.  The  simulation  revealed  that 
for  the  majority  of  the  system  configurations,  component 
dependency  caused  the  total  system  cost  variance  to  signif¬ 
icantly  deviate  from  its  value  under  conditions  of  component 
independence.  This  finding  indicates  that  weapon  system 
acquisitions  that  assume  component  independence  are  likely 
to  significantly  misestimate  the  total  system  cost  variance. 
The  impact  is  the  potential  misnllocation  of  scarce 
resources . 

There  were  several  trends  that  became  evident  when 
comparing  dependence  against  independence  using  the  cost- 
risk  assessment  method  developed  in  this  research.  The 
proportion  of  the  positive  cost  dependencies  in  a  system 
ci curly  had  the  s t o n g e a t  influence  over  the  difference  be¬ 
tween  tlie  dependent  and  independent  system  configurations. 
The  greater  the  proportion  of  positive  dependencies  in  the 
system,  the  greater  the  difference  between  the  total,  system 
cost  variances.  As  the  strength  of  the  cost  dependencies 
was  Increased,  the  difference  between  the  independent  and 
dependent  system  variances  Increased.  However,  the  increase 
was  small  relative  to  that  increase  which  occurred  when  the 


I 


proportion  of  positive  cost  dependencies  was  increased, 
Changes  in  the  size  of  the  system  did  not  appear  to  affect 
the  ratio  of  the  dependent  and  independent  variances.  How¬ 
ever,  the  researchers  believe  this  result  is  due  to  the 
limiting  assumption  of  sequential  component  dependencies. 

As  this  restriction  is  relaxed,  the  number  of  component 
dependency  pairings  increases  and  could  signif ica,ntly  affect 
the  ratio  of  the  variances.  Sensitivity  analysis  was  per- 
formed  on  the  method  to  evaluate  its  ability  to  predict 
total  system  cost  variance  under  conditions  of  Rho  squared 

estimation  error.  Results  show  that  the  method  predicts 

2 

total  system  cost  variance  fairly  accurately  when  R  estima- 

2 

tion  error  was  limited  to  .1.  As  the  R  estimation  error 
grew  to  .2,  the  method  did  not  predict  as  well.  The 
researchers  consider  a  .1  error  in  estimating  R  to  be  very 
good,  yet  even  this  small  amount  of  error  does  not  guarantee 
prediction  of  the  total  system  cost  variance.  When  the 
number  of  components  in  the  system  or  the  proportion  of  the 
positive  cost  dependencies  is  large,  the  predictive  ability 


j 

sJ 


of  the  cost-risk  assessment  method  is  not  very  accurate. 
However,  these  inaccuracies  may  be  tolerable  when  the  conse¬ 
quence  of  ignoring  component  dependence  is  considered.  The 
researchers  found  that  those  situations  where  the  variance 
was  difficult  to  predict  were  the  situations  with  the 
greatest  need  to  incorporate  dependence.  The  potential 
impact  of  each  error  would  have  to  be  weighed. 
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Limit- a  1 1  ons 


The  research  was  limited  in  several  ways.  Although 
subjective  assessments  of  component  dependence  will  always 
include  some  degree  of  error,  the  technique  used  to  quantify 
these  assessments  will  introduce  additional  error.  This  is 
particularly  true  for  the  ’descriptive  category'  technique 
described  in  Chapter  3.  The  analyst  has  wide  latitude  in 
assigning  Rho  values  to  each  category. 

The  simulation  only  tested  a  limited  number  of  system 
configurations,  Many  scenarios  that  occur  in  the  real  world 
were  Left  out.  In  particular,  wider  ranges  and  more  para¬ 
meter  settings  ore  needed  to  test  the  effects  of  negative 
component  dependence,  a  larger  range  of  rho  values,  and  the 
more  system  sizes. 

The  manner  in  which  the  Rho  values  were  generated  is 
also  a  weakness  in  the  research.  Both  positive  and  negative 
Rho  values  were  generated  from  the  some  distribution.  This 
assumes  that  the  magnitudes  of  both  the  positive  and  nega¬ 
tive  Rho  values  have  nn  equal  likelihood  of  occurrence.  In 
u  real,  system,  this  situation  Is  unlikely.  Kuther,  strong 
positive  relationships  may  be  more  likely  to  be  accompanied 
with  weak  negative*  ones  in  the  system. 

R  c  c:  u  in  in  o  n  d  a  1 1  o  n  g 

T'le  authors  feel  t h a t  further  research  should  be  con¬ 
ducted  w  i  t  h  this  cost-risk  assessment  methodology.  The 
following  ureas  are  recommended: 


1.  Sensitivity  analysis  should  be  performed  to  examine 
the  scenarios  that  were  not  examined  here.  A  clearer  iden¬ 
tification  of  those  situations  with  moderate  prediction 
capability  is  needed. 

2.  Batter  techniques  to  solicit  and  convert  subjective 
assessments  Into  measures  of  dependency  would  improve  the 
methodology.  The  use  of  conditional  probabilities  is  one 
possible  approach  to  consider. 

3.  Extension  of  this  methodology  should  be  applied  to 
a  WBS  hierarchy  to  compare  the  trends  which  were  discussed 


in  this  thesis. 


Appendix  A :  Calculation  of  Sample  Size 


Banks  and  Carson  describe  a  procedure  for  determining 
the  required  number  of  simulation  repetitions..  A  two*-sided  • 
confidence  interval  for  the  mean  of  a  distribution,  “x,  based 
on  the  Students  t  distribution  is  given  by}. 


x  +  [(t)  (s  /  r*)] 


( A .  1 ) 


where 


t  •#  critical  value  for  t  distribution 
s  -  sample  standard  deviation  of  x 
r  ■  number  of  simulation  repetitions 


The  precision  of  the  confidence  interval  is  specified  so 
that  it  estimates  the  true  mean  within  a  tolerable  error, 
+e  ,  and  with  a  desired  level  of  confidence.  Therefore, 
from  fiq  ( A  .  1 ) 


«  >  (t)  (s  /  r*) 


(A. 2) 


The  number  of  repetitions  required  to  achieve  an  estimate  of 
the  moan  within  +_t  accuracy  is  determined  by  choosing  an 
initial  number  of  repetitions.  The  required  number  of  repe¬ 
titions  is  then  solved  by  manipulating  Eq  (A. 2)  as  follows: 


t  >  (  t.  )  (  s  /  r  J  ) 

>  (t)  <s  /€  ) 


(A. 2) 
(A. 3) 


The  number  of  repetitions  is  calculated  using  the  standard 
deviation  from  the  inital  sample,  the  critical  t  value,  and 
the  desired  accuracy  (1:427).  Generally,  a  few  iterations 
of  Eq  (A. 4)  are  required  in  order  to  adjust  the  t  value  as 
the  degrees  of  freedom  change  with  the  number  of  repetitions 
that  is  calulated  (1:426-427). 

the  authors  selected  50  initial  repetitions  for  the 
five  component  system  and  100  for  the  fifty  and  one  hundred 
component ,  systems .  A  95%  confidence  level  seemed  reasonable 
and  the  amount  of  tolerable  error  was  subjectively  estab¬ 
lished  at  50,  roughly  half  the  sample  standard  deviation  for 
the  five  component  system  (s»110).  The  mode  for  the  tri¬ 
angular  distribution,  the  proportion  of  positive  rhos,  and 
the  rho  squared  estimation  error  were  set  to:  m«.707  , 

p*=.75  ,  and  e» .  1  .  The  'normal*  case  was  used  as  the 

basis  for  the  calculations. 

For  the  5  component  system,  solving  Eq  (A. 4)  revealed 
that  20  repetitions  were  sufficient. 

r  >  t ( t )  (s  /  «)]2  (A. 4) 

r  >  [ (2.02)  ( 110  /  50) ]2 

r  2  20 

However,  50  repetitions  were  selected  for  the  5  component 
system  in  order  to  provide  a  smoother  histogram.  From  Eq 


(A. 2),  Che  error  for  the  5  component  system  is  reduced  from 
50  to  31  when  r*50  .  The  sample  standard  deviations  for 

the  50  and  100  coiiinonent  systems  were  373  and  54  5,  respec¬ 
tively.  Applying  the  above  procedure,  the  calculated  sample 
eisses  were  213  and  456  for  n«50  and  n»100  ,  respec¬ 

tively. 

As  discussed  in  Banks  and  Carson,  the  model  is  run  with 
this  number  of  repetitions  and  adjustments  are  made  for  the 
new  sample  standard  deviation  (1:427).  The  results  of  these 
calculations  yielded  re  2  29  and  r ■  393  .  These  values 
were  rounded.  The  number  of  repetitions  used  in  this  re¬ 
search  is  shown  in  Table  A.l. 


Table  A.l 

Simulation 

Replications 

number  of 
system 
components 

number  of 

simulation 

repetitions 

5 

50 

50 

230 

100 

400 

Appenndix  B:  Simulation  Program 


The  following  section  of  the  Fortran  code  declares  these 
real  and  integer  variables  and  arraya: 

ARRAY  PURPOSE 

var  Stores  comppnent  variances 

sign  Stores  random  numbers  (uniformly  distributed  0-1) 

used  to  determine  the  sign  of  rho  values 

numb  Stores  random  numbers  (uniformly  distributed  Chi) 

used  to  calculate  magnitude  of  rho  values 

rl  Stores  low  Rho  for  component  pairs 

rn  ,  'Stores  normal'  Rho  for  component  pairs 

rh  Stores  hi'gh  Rho  for  component  pairs 

covl  v  Stores  cqvariance  computed  with  low  Rho 

covn  Stores  covariance  computed  with  normal  Rho 

covh  Stores  covariance  computed  with  high  Rho 

seeds  Stores  random  numbers  (uniformly  distibuted  0-1) 

to  be  used  as  random  number  stream  seeds 


VARIABLE 

C  0  V  S  1 

covsn 
c:  o  v  s  h 
va  rs 
t.  s  vl 
t  S  VII 
Is  vl) 

i  X 


IDENTITY 

Sum  of  low  covariances 
Sum  of  normal  covariances 
Sum  of  high  covariances 
Sum  of  component  variances 
Total  system  variance* -  low 
Total  system  variance-normal 
Total  system  vnrionce-high 
Initial  random  number  s  t  r  e  u  m  seed 


B.  1 
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iy 

yfl 
ay  f  1 


Dummy  variable  for  random  number  generation 
Random  number  generated  by  subroutine 


Dummy  variable  used  for  arithmetic  operations  on 
random  number 

m  Mode  of  triangular  distribution  of  rho  values 

e  Rho  estimation  error 

p  Proportion  of  positive  rho  values 

n  Number  of  system  components 

r  Number  of  simulation  repetitions  (Appendix  A) 

rnsq  Rho-normal  squared 

i  Counter 

k  Counter 


c  program  l.f 

real  var(100),sign(99),numb(99),rl(99),rn(99),rh(99), 
•)*covl(99),covn(99),covh(99)  ,  seeds (400)  ,covsl,covsn, 
^covsh,vars,y£l,tsvl,tsvn,tsvh,m,ayfl,rnsq,e,p 
integer  ix,iy,n,i,k,r 


i 


$ 


8 


£ 


The  following  segment  opens  a  file  where  the  output  of  the 
simulation  is  stored  and  sets  the  system  parameters  as 
described  in  Chapter  3,  Files  outl-out54  were  used. 


open  (6,file«'out37') 
rewind  h 

p-1.0 
e  =  .  2 
n«  1 00 
m« .  5 
r  =  400 


This  section  calls  the  random  number  generator  (subroutine 


B.  2 


RaNBU)  and  initializes  the  random  number  stream  seed. 


i  y  =  0 

ix- 83647 
yf  1-0. 0 

call  randu(ix,iy,yfl) 


This  loop  calls  the  RANDU  subroutine  to  generate  random 
numbprs  that  will  be  converted  to  a  different  5  digit  inte¬ 
ger  random  number  stream  seed  for  each  simulation  repeti¬ 
tion.  This  is  to  ensure  that  the  simulation  results  are  not 
autocorrelated  (1:391). 


do  20  i-l,r 
ix-iy 

call  r andu( ix , iy , yf 1 )  . 
ayfl-int(yfmOOOOO) 
seeds(i)«ayfl 
20  continue 


This  loop  calls  the  subroutine  to  generate  random  numbers 
that  are  used  for  component  variance  values  (0-100).  These 
variances  are  held  constant  for  all  iterations  of  a  given 
system  and  are  identical  for  all  simulations  of  equal 
component  size. 


do  30  i-l,n 
ix-iy 

call  randu(ix,iy,yfl) 
ayfl-yflw100 
var ( i  )  -ayf  1 
30  continue 


This  section  begins  the  loop  for  iterations  of  the  defined 
system  and  initializes  the  sum  of  variances,  covariances,  and 
total  system  variance  to  0. 


do  1000  k-1 ,  r 
vars-0. 0 
covsl*0. 0 
covnn-0 . 0 
co vsn*»0 . 0 
tsvl-0,0 
ts  vn*0 . 0 
ts vh«0, 0 


This  segment  initializes  the  random  number  stream  to  a  new 
value  for  each  repetition. 


iy*0 

ix«seeds ( k ) 
yfl-0.0 

call  randu(tx,iy,yfl) 


Thin  loop  loads  an  array  with  random  numbers  for  each  compo¬ 
nent  pair.  These  numbers  are  also  non-autocor related  and 
will  determine  which  component  pairs  have  negative  correla¬ 
tion, 


40 


do  40  i.  =  l  ,  ( n -  1  ) 
i  x  =  i  y 

call  randu(ix,iy,yfl) 
sign(  i  )  =» y  f  1 
continue 


The  following  loop  loads  an  array  with  random  numbers  for 
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each  component  pair.  These  numbers  will  be  used  to  calcu¬ 
late  non-autocorrelated  Rho  values  based  on  a  triangular 
distribution. 


do  50  1»1 , ( n-1 ) 
ix-iy 

call  randu(ix',iy,yfl) 
numb( i )-yf 1 
50  continue 


This  section  transforms  the  random  numbers  in  array  'numb* 
into  a  Rho  value  for  each  component  pair.  Banks  and  Carson 
use  the  cumulative  distribution  function  (c.d.f.)  to 
transform  the  random  number  into  a  random  variate  of  the 
triangular  distribution.  The  c.d.f.  for  a  triangular 
distribution  on  the  interval  (0,1)  is: 

x  <  0 
0  <  x  _<  m 
m  <  x  <_  1 

x  >  1  (B.l) 

where 

m  ■«  mode  of  t  r  iangular  di  s  t  r  i  but  ion 
x  ■  rho  value  on  the  distribution 

hlq  (B.l)  is  then  set  equal  to  the  random  number,  'numb'  for 
each  interval  of  the  random  variate.  For  0  <  x  <  m ; 


1°, 

x2  /  m, 

[1-  (1-xr]  /  ( 1  -m ) , 
1. 


n  u  m  b  *» 


m 


(13.2) 
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For  m  <  x  <  1 : 


numb 


[1-  (l-x^l  /  ( 1  -m ) 


(B.3) 


« 


Therefore,  Rho  values  (x)  in  the  interval  0  <_  x  <_  m  implies 
that  0  ,<  numb  £  m.  In  this  case,,  the  Rho  value  for  , random 
numbers  between  0  and  m  are  calculated  by  rearranging 
Eq  (B.2)  as  follows: 


(m)  (numb) 
t (m)  ( numb) ] 


i 


(  B .  4 ) 
( B .  5 ) 


% 


Rho  values  (x)  in  the  interval  m  <  x  <,  1  implies  that 
m  <  numb  _<  1,  The  Rho  value  for  random  numbers  between  m 
and  1  are  calculated  from  Eq  (B.3)  as  follows: 


(1-x)  /  ( 1  -  in ) 

2 


<  1-x)' 


1-  numb 
(1-m)  (1-numb) 


(1-x)  -  [ ( 1 - m )  (1-numb)] 


-x  » 


- 1  +  [ ( 1 -m )  ( 1-num  b  )  ] i 


x  -  1-  [(1-m)  (1-numb)] 


To  summarize  the  transformation: 


* 


[(tn)  (numb)], 

J -  |(l-m)  (1-numb)] 
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(B.6) 
(B.7) 
(  B .  8  ) 
( B  .  9  ) 
(B. 10) 


0  <  numb  _<  m 
m<numb<il  (B.ll) 
(1:158, 299-300) 


’/nmrTfvrnr  \r*  xrvvnrTJi*\rw\ni  wrw  vtith  wrw  u'u  wv  wnw*  wn  ui  iruw^knc  u  w  wrw^  wm  wwu«  siw*  wk  u  ▼  un  'jn»yi^Tu»w»w  «  ^  »  w 


V  "  U  •»  U  •  V 


do  60  i* 1 , ( n- 1 ) 

if(numb(i)  ,ge.  0,0  .and,  numb(i)  , 1 e .  m)  then 
rn(i)=sqrt(m'l|,numb(i)) 
else 

rn(i)-1.0-sqrt((1.0-m)#(1.0-numb(i))) 

60  continue 


This  loop  changes  the  Rho  to  a  negative  value  for  component 
pairs  when  'sign1  random  number  is  greater  that  the  identi¬ 
fied  proportion  of  positive  Rhos. 


do  70  i-l.(n-l) 

if  (sign(i)  ,gt.  p)  then 
rn(i)— rn(l) 
end  if 

70  continue 


The  next  loop  checks  the  Rhp  values  and  assigns  upper  and 
lower  Rho  limits  based  on  the  sign  of  rho  and  the  amount  of 
Rho  squared  estimation  error.  The  Rho  limit  which  produces 
the  lower  total  syHtem  variance  is  assigned  to  array  *  rl *  • 
the  other  limit  is  assigned  to  array  'rh'.  Rho  limits  that 
are  outside  the  interval  (0,1)  are  truncated  to  the  appro¬ 
priate  limit,  0  or  1.  An  example  of  this  procedure  is  given 
in  Chapter  3. 


do  80  i-1 , (n-1 ) 

if  (rn(i)  .ge.  0.0)  then 
rnsq-rn(  i  )<M,,2 

if  (0,0  ,gt.  (rnsq-e/2))  then 
r 1 ( i ) -0 . 0 

rh(t)»sqrt(rnsq+e/2) 
else  if  ((rnsq+e/2)  ,gt.  1,0)  then 
r 1 ( i ) -sqr t ( r nsq-c/2  ) 
rh(i)«1.0 


else 


rl(i)«sqrt(rnsq-e/2) 
rh(i)»sqrt(rnsqte/2) 
end  if 

else  if  (rn(i)  .It.  0.0)  then 
rnsq  =  rn(i)w4f2 

if  (0.0  ,gt.  (rnsq-e/2))  then 
rh( 1 )-0. 0 

rl(i)»-sqrt(rnsq+e/2) 
else  if  ((rnsq+e/2)  .  gt .  1.0)  then 
rh(i)«-sqrt(rnsq-e/2) 
rl(i)  —  1 .0 

else 

rh(i)--sqrt(rnsq-e/2) 
rl(i)«-sqrt(rnsq+e/2) 
end  if 

end  if 

80  continue 


The  following  section  uses  the  normal,  low,  end  high  rho 
values  for  each  component  pair  and  their  associated  vari¬ 
ances  to  calculate  the  normal,  low,  and  high  covariance 
values . 


do  90  i-l,(n-l) 

covl(i)«rl(i)<|,sqrt(var(i)wvar(i  +  l)) 
covn(i)»rn(i)<(,sqrt(var(i)wvar(i  +  l)) 
covh(i)»rh(i)wsqrt(var(i)#var(i+l)) 
90  continue 


This  loop  sums  the  variances  for  all  system  components. 


do  100  i«l,n 

vars-vars+ var ( i ) 
100  continue 


This  loop  sums  the  normal,  low,  and  high  covariances  for  all 
component  pairs. 
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•S'r  >7. 


do  110  i-l.(n-l) 

covsl«covsl+covl(i) 
covsn+covsn+covn(i) 
covsh=covsh+covh(i) 
110  continue 


This  segment  calculates  the  normal,  low,  and  high  total 
system  variance.  Covariance  sums  are  multiplied  by  2  as 
discussed  in  Chapter  3.  The  output  is  written  to  the  file 
identified  in  the  OPEN  statement  at  the  beginning  of  the 
program.  Program  execution  loops  back  for  another  repeti¬ 
tion  of  the  same  system  until  r  repetitions  have  been  per¬ 
formed  , 


tsvl«vars+2#covsl 
tsvn«vars  +  2'l,,covsn 
tsvh«vars  +  2'M'covsh 

write( 6 , 7) ts vl , tsvn , ts vh 
7  format (3(f6.0,x)) 

1000  continue 


After  the  final  repetition  is  complete,  the  output  file  is 
closed  and  program  execution  halts. 


close  (6) 
end 


The  RANDU  subroutine  generates  all  random  numbers  required  by 
the  simulation. 


Appendix  C:  BMDP  Programs 


The  University  of  California  at  Los  Angeles  (UCLA)  BMDP 
statistical  package  was  used  to  provide  the  histograms  and 
the  sorting  procedure  that  aided  the  data  analysis  (6:43, 
74-78,  112-113,  124-132).  The  following  programs  were  used 
to  perform  their  respective  functions. 

Program  1:  Histograms 

The  variable  names  tsvl,  tsvn,  and  tsvh  correspond  to 
those  used  in  the  simulation  program  discussed,  in 
Appendix  B. 

/problem  title  is  'run54'. 

The  problem  paragraph  states  the  title  of  the  BMDP 
execution  program.  In  this  case  the  name  of  the  program  is 
r  un  54 . 


/input.  variables  ore  3 . 

format  is  '^(fG.OjX))'. 


This  portion  of  the  program,  the  input  paragraph,  states  the 
number  and  format  of  variables  to  be  read  in  ns  data. 


/variable  names  are  tsvl , is vn , tsvh  , 

b 1  a  n  k  u  ore  % o  r  o  . 


The  variable  paragraph  names  the  variables  so  that  they 


may  be  identified  in  the  output.  The  second  line  of  code 
makes  the  package  read  all  blank  spaces  in  the  data  as 
zeroes. 


/group  cutpoints  (1)  *  2300  to  5800  by  175, 

cutpoints  (2)  *2650  to  6150  by  175. 
cutpoints  (3)  -  3175  to  6675  by  175. 

The  group  paragraph  sets  the  intervals  for  the 
histograms.  (1)  represents  tsvl,  (2)  represents  tsvn,  and 
(3)  represents  tsvh.  The  high  and  low  values  were 
determined  by  the  sort  program  to  be  discussed  below.  A 
value  consistent  within  each  run  is  used  to  specify  the 
width  of  each  interval.  In  the  case  of  run54,  175  is  the 
interval,  width. 


/plot  type  is  hist. 

scale  is  0,3. 

The  plot  paragraph  specifies  that  a  histogram  will  be 
plotted ,  The  second  line  indicates  that  zero  is  the  low 
value  on  the  histograms  and  that  each  'X'  on  the  histogram 
represents  three  data  points. 


/  v  n  d 

This  ends  the  program.  After  this  statement,  the  data 
is  listed  in  the  format  described  in  the  input  paragraph. 


Program  2:  Sorting 

This  program  is  the  same  as  the  histogram  program  with  some 
changes.  The  problem  and  variable  paragraphs  remain 
the  same.  The  following  code  is  added  to  the  input 
paragraph  . 

/input  variables  are  3. 

format  is  ' ( 3 ( f 6 , 0  ,  x  ) )  '  . 
sort  is  tsvn,  tsvl,  tsvh. 

The  sort  statement  has  the  tsvn  variable  name  first  to 
give  it  precedence  in  the  sort  since  it  is  the  critical  data 
column . 

Instead  of  a  plot  paragraph,  a  print  paragraph  was  used 
to  display  the  sorted  data. 

/print  data. 

order. 


Appendix  D:  Histograms  for  RUN1-RIJN54 

Th^s  Appendix  contains  the  BMDP- generated  histograms 
for  RUN1-RUN54.  The  histograms  show  the  'low',  'norma1 
and  'high'  distributions  of  total  system  cost  variance  for 


each  RUN  number 
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Appendix  Es  Overlap  Percentages  for  Variance  Distributions 

The  percentages  in  the  table  represent  the  percent  of 
t}ie  observations  that  were  above  the  percentile  for  the 
'low'  distribution  and  the  percent  of  the  observations  that 
were  below  the  percentile  for  the  'high*  distribution. 
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RUN  19  20  21  22  23  24  25  26  2  7  RUN19  20  21  22  23  24  25  26  27 


05  3  61  69  7  73  76  16  79  79  95  3  57  72  10  65  77  17  71  80 

10  2  43  56  4  54  66  7  53  73  90  0  47  53  4  57  63  10  60  70 

15  2  30  43  3  44  59  6  50  63  85  0  33  41  1  51  54  4  53  62 
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05  38  84  83  51  87  88  60  88  88  95  47  80  85  51  86  90  58  88  90 
10  31  70  76  43  77  78  49  80  80  90  32  69  76  41  73  77  46  78  79 
15  23  60  68  35  66  75  43  68  76  85  25  58  69  31  62  74  36  67  76 
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